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STEM Learning Opportunities Providing Equity (SLOPE): An Investing in Innovation (i3) Grant 

Final Evaluation Report 


Introduction 

In 2010, Dr. Penny Edgert, on behalf of the California Education Round Table Intersegmental 
Coordinating Committee (CERT ICC) and the Alliance for Regional Collaboration to Heighten Educational 
Success (ARCHES), was awarded a five-year Investing in Innovation (i3) development grant from the U.S. 
Department of Education. In partnership with ConnectEd: The California Center for College and Career, 
Dr. Ivan Cheng (California State University, Northridge), and R.T. Fisher Educational Enterprises, Principal 
Investigator (P.l.) Dr. Edgert and her grant team (hereafter referred to as the Intervention Team) 
implemented an intervention entitled Science, Technology, Engineering, and Mathematics (STEM) 
Learning Opportunities Providing Equity (SLOPE). 

WestEd served as the external evaluator for this grant. Under the direction of Dr. Carole Gallagher and 
lead analyst Dr. Kevin Huang, a team of WestEd researchers conducted an Institutional Review Board 
(IRB) regulated, mixed-methods evaluation that included a randomized controlled trial (RCT). Oversight 
of many evaluation activities was provided by the i3 national evaluator, Abt Associates, who reviewed 
and approved the design, methodology, and planned analyses for the impact and implementation 
studies. The WestEd team held monthly conference calls with its i3 national evaluator representative to 
monitor study progress and discuss emerging challenges. The intent of this collaboration was to ensure 
that all i3 evaluations were conducted in compliance with the What Works Clearinghouse principles for 
rigorous scientific research. 

During the evaluation, quantitative and qualitative data were collected and analyzed to examine (a) the 
impact of the intervention on students and teachers and (b) the fidelity of implementation of the 
intervention components by teachers. Findings were intended to be used formatively by the P.l. and her 
team for intervention refinement prior to the RCT and summatively for decision-making about the 
effectiveness of the intervention based on findings from the RCT. This final report, developed for 
Dr. Edgert and her team, describes the methods used to conduct the evaluation and summarizes 
findings from key analyses completed during and following implementation of the intervention. 

Intervention Components and Time Line for Implementation 

The purpose of this intervention was to increase the success of all students, especially those with a 
history of low performance in mathematics, in grade 8 Algebra I through rigorous and relevant project- 
based contextual study that promoted learning of fundamental math concepts. 1 This curriculum was 
intended to help bridge the gap between the Algebra I content students learn in class and the associated 
college preparation leading to STEM careers. Intervention developers relied on research that indicates 
learning mathematics in a project-based context assists in promoting student motivation and interest 
and transference of skills by linking classroom activities with real-world mathematics. In keeping with 
the intervention's logic model (see Appendix A), the study curricula were supplemented with academic 
coaching opportunities for teachers to promote confidence with teaching the challenging algebra 
content in the California State Standards (the Common Core State Standards) and incorporating project- 


1 Algebra I is widely considered a "gatekeeper" course, one that is particularly important for success in STEM 
careers. 
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based learning as a forum for practical application of the knowledge and skills learned during formal 
instruction. 


The following activities took place during the five years of this grant: 

• Between the grant award in October 2010 through June 2012, the intervention components 
underwent various stages of development, review, and piloting. During that time period, the 
Intervention Team recruited 56 California grade 8 Algebra I teachers to participate in the two- 
year intervention (see the Summary of Findings section for descriptive information about this 
sample of teachers). 2 WestEd researchers randomly assigned these teachers to a Treatment or 
Control group. Teachers assigned to the Treatment group were trained to implement the 
intervention during the 2012-13 and 2013-14 school years. 3 All Treatment and Control teachers 
were required to complete a survey about their professional background (see Appendix B). 

• During the 2012-13 school year, the intervention was formally studied via a randomized 
controlled trial (RCT). During that time, the grade 8 Algebra I teachers in the Treatment group 
were exposed to the intervention components— strategic professional development 
opportunities and access to curricular materials that reinforced critical elements of standards- 
based mathematics instruction and provided information about STEM career pathways. All 
students assigned to grade 8 Algebra I classrooms taught by these teachers were exposed to the 
special curricula in addition to the district-specific curriculum. A small subset of students 
assigned to these classrooms also was exposed to an optional pre-grade 8 summer session (half- 
day on Monday-Friday for four weeks) taught by a trained Treatment teacher that was designed 
to support students identified as having high academic need. All students assigned to grade 8 
Algebra I classrooms taught by teachers assigned to the Control group were exposed only to the 
district-specific curriculum and were not invited to the summer session. All students— those in 
classrooms taught by teachers in both the Treatment and Control groups— were administered 
the end-of-grade, standardized state test in mathematics. Scores from that test and from the 
prior school year (grade 7) were collected for all students in classrooms taught by teachers in 
the Treatment and Control groups. 4 Information about implementation was collected from 
teachers via detailed surveys that required them to carefully document the ways in which they 
carried out the intervention. 


2 All districts targeted for recruitment had existing relationships with the ICC/ARCHES or ConnectEd through other 
types of activities. These districts were representative of the diversity of California geographically and 
demographically. All of the districts had middle schools that enrolled a majority of minority/low-income students. 

3 It is important to note that during both years of the intervention (2012-13 and 2013-14), Grade 8 students in 
participating schools were assigned to Algebra I classrooms using normal district-approved methods; those 
assigned to classrooms assigned to teachers in the Treatment group would be exposed to the intervention in 
addition to the district curriculum, and those assigned to teachers in the Control group were exposed only to the 
District curriculum. In this way, two cohorts of students would benefit from the intervention, those in grade 8 
during the 2012-13 school year (Student Cohort 1) and those in grade 8 during the 2013-14 school year (Student 
Cohort 2). 

4 During all phases of work, using IRB-approved methods and tools, obtaining informed consent was a priority for 
the evaluation team. In all districts except one, an opt-out procedure was used to ensure all students and their 
families were fully informed about their schools' participation in the study, how student-level data (test scores for 
grades 7 and 8) would be used, and who would have access to those data. In one district, parents were required to 
return a consent form indicating that their students' data could be used in the evaluation. 
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• During the 2013-14 school year, though the formal impact study was concluded, exploratory 
work continued under the same RCT conditions previously described. Teachers in the Treatment 
group again taught the intervention curricula in addition to the district-specific curriculum and 
information about implementation was collected. Teachers in the Control group taught the 
district-specific curriculum only. All students assigned to grade 8 Algebra I classrooms taught by 
teachers in the Treatment group were exposed to the special curricula in addition to the district- 
specific curriculum. A small subset of students assigned to these classrooms also was exposed to 
the pre-grade 8 summer session. All students assigned to grade 8 Algebra I classrooms taught by 
teachers assigned to the Control group were exposed only to the district curriculum. Because 
the Governor of California suspended state testing in Spring 2014, scores could not be collected 
for students using procedures used in School Year 2012-13. The Intervention Team, however, 
was able to secure permission from the state testing vendor to administer the test in those 
districts that would allow this supplementary administration. 5 

• During the 2014-15 school year, data collected during the 2012-13 and 2013-14 school years 
were analyzed. In addition, supplementary exploratory analyses were conducted to address 
specific research questions from the Implementation Team. Teachers assigned to the Control 
group were provided access to the intervention curricular materials. WestEd evaluators and the 
Intervention Team Director, Ms. Sharon Twitty, presented preliminary findings at conventions 
that included the annual meeting of the American Educational Research Association (AERA)(see 
Appendix C for the paper presented at AERA). 

Evaluation Approach 

A key goal of the evaluation of this i3 grant was to implement a research-supported process for 
meaningfully interpreting findings from an RCT using contextual information about implementation 
fidelity that was systematically collected during the course of an evaluation. In theory-based 
evaluations, the systematic examination of implementation of fidelity allows experimental-design 
researchers to open up the "black box," i.e., to better understand the causal processes underlying the 
outcomes that emerge (Donaldson & Lipsey, 2006). Specifically, a fidelity of implementation index 
describes the degree to which key components of the intervention are delivered with integrity, through 
adherence to the developers' intent and/or in keeping with the original program design (O'Donnell, 
2008). Contextual information that compares actual implementation processes and practices to ideal 
practices can provide valuable insight into possible explanations for statistically significant or non- 
significant findings (Century, Rudnick, & Freeman, 2010; Hulleman & Cordray, 2009; Mowbray, Holter, 
Teague, & Bybee, 2003; National Research Council, 2004). For evaluations focused on measuring the 
effectiveness of innovative curricular tools, this information may be particularly useful, as results can be 
used formatively by intervention developers to refine processes or procedures prior to scale-up 
(O'Donnell, 2008). 

This study uses an a priori intervention model, in which expectations about implementation are 
specified by the developers and documented by the evaluator prior to data collection (Hulleman & 
Cordray, 2009). These expectations are linked to the intervention's theoretical foundation (theory of 

5 Only about half of the total pool of participating schools agreed to this supplementary administration. WestEd 
researchers collected student responses, scored the responses, and completed the raw-to-scale score conversion 
using vendor-approved methods. In completing these activities, the evaluation team complied with all IRB 
regulations in place to protect the confidentiality of individual students and ensure data security. 
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action or logic model) and become the standard ("threshold") for fidelity for all study participants. As 
shown in the intervention logic model (see Appendix A), the inputs include curricular materials, training, 
and professional coaching, which support delivery of the instructional units at the classroom level during 
the summer prior to grade 8 and in grade 8. The student-level outcome of interest is performance on 
the state's standardized test for grade 8 in Algebra I, administered annually in late spring. 
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Methodology 

This evaluation was designed to (a) examine the impact of an intervention aimed at increasing the 
academic achievement of students in Algebra I, a "gatekeeper" course for careers in the STEM 
pathways, and (b) better understand the relationship between the impact of an intervention and 
implementation fidelity, as measured by participants' capacity to meet or exceed thresholds for 
performance set by intervention developers. The components of the intervention that were evaluated 
included (a) the Algebra I curricula (three project-based, drop-in instructional modules incorporated into 
the district-specific Algebra I curriculum plus the college-awareness activities), (b) the optional summer 
curricula, and (c) professional coaching activities for teachers. 

a) Algebra I Curricula. The Algebra I instructional units consisted of three project-based STEM- 
oriented academic units designed to be taught at various points during the school year and 
integrated into the district-selected mathematics curriculum. These units were entitled Puzzle 
Cube, Air Traffic Control, and Catapult Game. Each required the students to work in small teams 
to apply mathematical concepts to design or build a structure or tool. Overall, these three units 
were expected to take approximately forty class periods, with some flexibility in instructional 
time in order to accommodate each district's pre-existing guidelines for timing and sequence of 
instruction. Only teachers assigned to the treatment condition received these curricular 
materials. All students who were enrolled in Algebra I classrooms taught by teachers assigned to 
the treatment condition were exposed to the district's curriculum as well as to the intervention 
curriculum. Students taught by control teachers or teachers not participating in the study were 
exposed only to the district curriculum and did not have access to the intervention curricular 
materials. 

b) Optional Summer Curricula. Summer C.A.M.P. was a four-week program consisting of three 
project-based, STEM-oriented academic units that required creative problem solving and 
reinforced mathematics concepts and skills needed for success in Algebra I. For example, one 
unit required students to model and build wind turbines that could generate a given amount of 
power. This unit reviewed the concepts of fractions, angles, and constraints. Summer C.A.M.P. 
was designed for students who had not reached proficient status on the grade 7 state test in 
mathematics in spring 2012. In some districts, however, administrators were not able to fill all 
program slots with low-performing students; in those districts, some of the students who were 
invited to attend Summer C.A.M.P. had reached proficient status on the grade 7 state test in 
mathematics. In all districts, student participation was optional. Students assigned to classrooms 
taught by control teachers (i.e., teachers not participating in the study) were not eligible to 
participate in Summer C.A.M.P. To accommodate the addition of the supplementary summer 
session for a small subset of participating Algebra I students, WestEd researchers examined the 
impact of this intervention in "dosages" administered. That is, all students assigned to an 
Algebra I classroom taught by a treatment teacher were exposed to at least one dose of the 
intervention (project-based Algebra I curriculum in grade 8); a small subgroup of students also 
was exposed to an additional dose (summer session in addition to the Algebra I curriculum). 

c) Professional Coaching Activities for Teachers. Teachers assigned to the Treatment group were 
required to attend all scheduled professional development and coaching sessions focused on 
implementing the instructional units as intended and reinforcing effective instructional 


5 




strategies. Each was assigned a coach with whom they conferred on a regular basis and could 
discuss emerging challenges. Through the 2012-13 and 2013-14 school years, this included 
weekly coaching and collaborative meetings to strengthen teacher understanding of the 
mathematics content and increase the likelihood of consistent implementation across teachers. 6 

Impact Study 

As previously described, during the 2012-13 school year, the impact of the intervention was studied 
through a RCT. The research questions guiding the impact evaluation were as follows: 

• Contrast 1: On average, among students in participating grade 8 Algebra I classrooms in the 
2012-13 school year who scored at the proficient or above level on their grade 7 standardized 
assessment in mathematics, does performance on the state's grade 8 standardized assessment 
in mathematics (Algebra I) differ between students who are exposed to the grade 8 Algebra I 
intervention curricula and students who are not exposed to the Algebra I intervention curricula? 

• Contrast 2: On average, among students in participating grade 8 Algebra I classrooms in the 
2012-13 school year who scored at the below proficient level on their grade 7 standardized 
assessment in mathematics, does performance on the state's grade 8 standardized assessment 
in mathematics (Algebra I) differ between students whose are exposed to the grade 8 Algebra I 
intervention curricula plus Summer C.A.M.P. and students who are not exposed to the Algebra I 
intervention curricula? 

• Contrast 3: On average, among all students participating in this evaluation study with a score 
from the grade 7 state tests in mathematics administered in 2012, does performance on the 
state's grade 8 standardized assessment in mathematics (Algebra I) differ for students who are 
exposed to any intervention components and students who are not exposed to any intervention 
components? 

In all cases, the contrast of interest for the impact study was performance on the state's standardized 
Algebra I test by students assigned to Algebra I classrooms taught by treatment teachers compared to 
performance by students assigned to Algebra I classrooms taught by control teachers. Analyses were 
conducted using a two-level hierarchical linear model (students nested within teachers) that examined 
the impact of the intervention on test scores. A cross-tabulation of cohort by demographic variables 
(e.g., gender, free/reduced lunch status, race/ethnicity, and English learner status) also will be 
presented. 

A statistical power analysis was conducted to determine the minimum detectable effect size for the first 
two contrasts. Findings were as follows: 

• For student-level analyses for Impact Research Question #1, the MDES = 0.24 (MDES = 0.28 if 30 
teachers per group, holding all others constant), assuming that: (a) two sections per teacher and 
30 students per section, (b) 30 teachers per group (T or C), (c) intra-class correlation = 0.2, (d) 
teacher-level covariates explain 50% of variance on student outcome, and (e) alpha = 0.05 & 
power = 0.8. 

6 Participating teachers received the following incentives to participate, regardless of group assignment, once they 
completed all required activities: $750/year + iPad2. Treatment teachers who taught a summer session could earn 
an additional $550 per session. Control teachers did not receive any coaching or professional development from 
the SLOPE Intervention Team, hence their incentives were awarded following completion of the background 
survey and semi-annual check-ins with the Intervention Team's director. 
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• For student-level analyses for the Impact Research Question #2, the MDES = 0.27 (MDES = 0.31 
if 30 teachers per group, holding all others constant). 

Implementation Fidelity Study 

The goal of the implementation study was to better understand the ways in which teachers assigned to 
the Treatment group carried out the required components of the intervention and the choices teachers 
made during the course of the study. Such information, collected from each participating teacher and 
then aggregated up to the program level, can shed light on how fidelity of implementation affected the 
student learning outcomes as specified in the logic model. Analyzing these data can promote more 
meaningfully interpretations of findings from the impact study. 

Core activities for teachers in the SLOPE Treatment group included use of the prescribed curricular 
materials during Algebra I instruction, attendance at training activities, and participation in coaching 
sessions. These components were mandatory, but within each component, teachers could make 
individual choices about certain elements of the activities. For example, each drop-in unit was divided 
into lessons, and lessons were subdivided into steps (e.g., worksheets, activities, instructional modules). 
Teachers were not required to use every lesson in a particular unit to reach the threshold for fidelity. In 
addition, those teachers who were assigned to the Treatment group and who elected to teach a summer 
session were required to use prescribed mathematics curricular materials during instruction and to 
attend a special training session. Actions taken and decisions made were collected via an 
implementation log or survey (see Appendix D for example survey) that teachers in the Treatment group 
completed that documented the ways in which they implemented the various components of the 
intervention (e.g., time spent on each unit). 

The method used in this study to evaluate fidelity of implementation included the following steps: 

1. Specification of the intervention logic model (theory of action) that identifies all key components 
of the intervention, the mediating factors through which the intervention is implemented, and the 
outcomes that the intervention is designed to achieve (see Appendix A); 

2. Identification of the instruments (e.g., surveys, attendance rosters, and other documentation) 
that will be used to collect information about each teacher's experiences; 

3. Development of a matrix that operationalizes the constructs of interest in terms of the 
observable indicators that are collected and that can be used to evaluate the degree to which 
each component was implemented "with fidelity," or in accordance with the developers' intent 
(see Appendix E for an excerpt); and 

4. Calculation of a composite fidelity index for each component (e.g., training, coaching, or 
curriculum unit) for each teacher (teacher-level score) and for the entire sample (program-level 
score). 

This method calls for clear linkage among the intervention logic model, which explains the hypothesized 
cause for the differences in outcomes between the treatment and control conditions, and the selected 
fidelity indices. Importantly, it also calls for the thresholds for fidelity to be determined by the 
intervention developers, with support from the evaluator. This methodology promotes study coherence, 
ensures that findings that emerge from the implementation fidelity analyses are meaningfully associated 
with the findings from impact analyses, and supports the validity of findings that emerge from the 
implementation study. 
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Consistent with this method, with support from the i3 National Evaluation team, WestEd evaluators 
designed a fidelity matrix (see Appendix E for an excerpt) that is linked to the logic model and aims to 
capture meaningful and discernible levels of implementation among teachers. 

Exploratory Analyses 

Additional research questions were explored in the 2014-15 school year. Findings from associated 
analyses were intended to provide feedback that the Intervention Team could use formatively to refine 
the intervention for application beyond this grant. These findings were not used for formal reporting of 
the impact or implementation study results, but are included in the findings section for review by the 
Intervention Team. This work explored the following research questions: 

• Was participation in the summer program consistent with the program's goals? Specifically, 
were any students who attended the summer program not enrolled in a Treatment teacher's 
classroom during the academic year? Were any students who attended the summer session 
enrolled in a Control teacher's classroom during the academic year? And, overall, how many 
students participated in the summer program? 

What can we conclude about how implementation affected outcomes? Specifically, was there a 
relationship between the degree to which participating teachers implemented the intervention as 
intended and student-level outcomes? 
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Summary of Findings 

Impact Study Findings 

As previously described, the contrast of interest for the impact study was performance on the state's 
standardized Algebra I test by students assigned to Algebra I classrooms taught by Treatment teachers 
compared to performance by students assigned to Algebra I classrooms taught by Control teachers. 
Analyses were conducted using a 2-level hierarchical linear model that examined the impact of the 
intervention on test scores. 

The final analytic sample for the 2012-13 cohort included 1,384 students assigned to 28 Treatment 
teachers and 1,088 students assigned to 27 Control teachers. 7 Table 1 presents descriptive information 
about the teacher participants. 

Table 1. Descriptive Information About Teacher Cohort, 2012-13 School Year 



n 

Percentage 

Male 

Average Years Mathematics 
Teaching Experience 

Percentage with 
Degree Beyond 
Bachelor's 

Treatment 

28 

39% 

10.86 

32% 

Control 

27 

11% 

8.52 

30% 

Total 

55 





Students who were included in the 2012-13 cohort met the following criteria: (1) were enrolled in an 
Algebra I class taught by a participating treatment or control teacher; (2) were classified as a grade 8 
student by the district; (3) had a valid test score for grade 8 Algebra I; and (4) did not opt out of data 
collection (most districts) or had parent permission to participate in data collection (one district). 8 All 
other students were included for instruction but excluded for impact analyses. 


Preliminary findings from the analyses are provided in Table 2. 

Table 2. Research Questions and Planned Analyses for Impact Study (RCT Year 1) 


Research Question 

n 

Mean Test Score 

T/C Mean Difference 
(S.D.) 

p-Value 

Contrast 1 

T= 673 

C = 519 

T = 351 

C = 357 

-6.01 

(4.798) 

.210 

Contrast 2 

T= 149 

C = 452 

T = 282 

C = 288 

-5.73 

(3.336) 

.086 

Contrast 3 

T = 1,384 

C = 1,088 

T = 320 

C = 325 

-5.38 

(3.953) 

.174 


^significant at .05 level 


7 Eight teachers were not eligible to participate in the study, as they were not assigned to teach grade 8 Algebra I in 
fall 2012. The remaining teachers, from the original cohort of 70, exited the study for personal reasons. Please 
see the Sample Description (CONSORT) table in Appendix F. 

8 As previously described, the majority of schools/districts had an "opt-out" policy that involves informed consent 
but does not require a parent to return a permission form to the school. One district, however— the largest 
district in the study— had an active consent policy requiring a student's guardian to return a permission form in 
order to collect data about that student. 
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As shown in Table 2, during the 2012-13 school year, no contrast showed a statistically significant 
difference at the .05 level. Students who were assigned to classrooms taught by T reatment teachers did 
not perform differently than those assigned to classrooms taught by Control teachers. 

Implementation Fidelity Study Findings 

As shown in the sample survey in Appendix D, we describe the levels of implementation that the 
developers, in consultation with the research team, set as the thresholds for fidelity. Appendix B also 
includes teacher-reported information about the number of lessons used, the time spent on each 
lesson, the steps used within each lesson, and the estimated amount of effort needed for each lesson. 
The indexing scheme included in Appendix B provides an overall assessment of an individual teacher's 
fidelity to the implementation model. 

A sample of findings from the implementation study for Algebra I teachers (the coaching component 
and Algebra I drop-in units) is presented in Table 3. Findings for other components (distribution of 
curricular materials, participation in professional development seminars, and implementation of College 
Awareness Curriculum activities) are provided in Appendix G. Thresholds for program-level fidelity (80% 
or more of teachers meeting individual thresholds for each component) also are indicated. 

Table 3. Preliminary Implementation Study Findings, by Component 


Component 

Description 

Number of 

Teachers 
Meeting Fidelity 
Threshold 

Percent of 

Teachers 
Meeting Fidelity 
Threshold 

Program- 

Level 

Fidelity 

Coaching 

Teacher participates in an average of 
one or more hours of coaching and 
collaboration sessions for each unit. 

16/28 

57% 

No 

Unit 1: 

Puzzle Cube 

Teacher implements Unit 1 of 
intervention curriculum. 

25/28 

89% 

Yes 

Unit 2: 

Air Traffic Control 

Teacher implements Unit 2 of 
intervention curriculum. 

18/28 

64% 

No 

Unit 3: 

Catapult Game 

Teacher implements Unit 3 of 
intervention curriculum. 

21/28 

75% 

No 


As shown in Table 3, in terms of the coaching component, while every teacher participated in some level 
of coaching, only 16 of the 28 Algebra I teachers participated regularly enough and for enough time to 
reach the threshold for fidelity. The implementation expectations for each of the core elements of this 
intervention, the three drop-in units, are reported separately. The percentages of teachers reaching the 
fidelity threshold ranged from 64% for Unit 2 to 89% for Unit 1. Teacher level data showed that only 16 
of the 28 treatment teachers met the threshold for fidelity for all three units. 

Figure 1 (next page) shows the trends in instructional time spent on each lesson. The marks 
corresponding to the graph's left axis in Figure 1 show the average percentages of expected time that 
teachers spent on each lesson within the Algebra I drop-in units. 
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Figure 1. Percentages of expected time spent on each lesson 
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As an example, the curriculum developers planned for the first lesson of the first unit ("Puzzle 1") to 
take 75 minutes of class time. If the average implementation time for teachers who reported using this 
lesson at all were exactly 75 minutes, the mark would be exactly at 1. As shown in Figure 1, the average 
implementation time was 101% of the expected time. If the average were below 100%, the mark would 
fall below 1. The bars corresponding to the graph's right axis show the numbers of teachers who did not 
implement a given lesson at all. A taller bar indicates that more teachers did not implement any steps in 
the given lesson. Treatment teachers appear to start with high levels of implementation for each unit, 
but drop-offs in implementation levels are increasingly evident toward the ends of units, especially for 
the second and third units. 

Figures 2a and 2b (next page) show the medians and interquartile ranges of the data that are shown in 
Figure 1. The data in these figures are divided into two subsamples of treatment teachers: treatment 
teachers who implemented all of the curriculum units with fidelity (Figure 2a) and those who did not 
implement one or more of the curriculum units with fidelity (Figure 2b). 
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Figure 2a. Teachers implementing all curriculum units with fidelity 



Percentage of Expected Time Used for Each Lesson by "High- 
Implementing" Teachers 


2.5 
2 

1.5 
1 

0.5 

0 


N 

3 

Oh 


CN CO 
O) 

N N 
N N 
3 3 

Oh Oh 


LD 

01 

N 

N 

3 

Oh 


CN CO 
u u 


< < 


03 03 

S-, S-, 


X 


LO VO 
U U 


03 03 

S-h ^ 


E-h 

S-, 

< 


H N CO ^ 


3 =3 3 

Oh Oh Oh 

03 03 03 

4-> 4-J 4-> 

03 03 03 

U U U 


O O 

Oh Oh 
03 03 

■4—* 4-» 

03 03 

U U 


Median 

First Quartile 

Third Quartile 


Figure 2b. Teachers who did not implement one or more curriculum units with fidelity 



As shown in Figure 2a, teachers who implemented all three curriculum units with fidelity generally had a 
median amount of time spent on each lesson at or just above the expectation. In contrast, as shown in 
Figure 2b, teachers who did not implement every unit with fidelity had much more variation in the time 
spent on implementation across lessons. 

Among the 28 treatment teachers, 16 appear to have completed all three required units with a high 
level of fidelity, while 12 completed the units with a mix of high and low levels of fidelity. To explore the 
effect of these different rates of implementation in greater depth, we conducted an impact analysis that 
was similar to Contrast 3 (all treatment and control students included), examining three teacher groups: 
treatment teachers with a high level of fidelity on all three units ( n = 16) vs. other treatment teachers ( n 
= 12) vs. control teachers (n = 24). We chose to focus our investigation on the curricular units because 
they are the core components of the intervention. The Contrast 3 sample was used so that there would 
be more students in each of the two newly constructed treatment groups, and thus more statistical 
power to detect possible differences. 9 


9 It should be noted, however, that the study was not powered to detect such group differences. Non-significant 
differences could be primarily due to an insufficient number of teachers in comparison groups. 
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Table 4 summarizes the findings from this analysis. 

Table 4. Impact Analysis on Student Algebra I Scores, by Three Teacher Groups 


Group Membership 

n 

Adjusted Mean 

Mean Difference 

p-Value 

Effect Size 

Treatment (low fidelity) 

581 

311.62 




Treatment (high fidelity) 

803 

324.65 




Control 

1,088 

324.62 




Diff: T-low and C 



-13.00 

0.039* 

-0.21 

Diff: T-high and C 



0.03 

0.996 

0.00 

Diff: T-low and T-high 



-13.03 

0.048* 

-0.21 


^significant at .05 level 


As shown in Table 4, students with high-fidelity treatment teachers did not perform differently from 
control students on the Algebra I assessment. Students with low-fidelity treatment teachers, however, 
scored significantly lower than control students. Similarly, students with low-fidelity treatment teachers 
also scored significantly lower than students with high-fidelity treatment teachers. 

Findings from Exploratory Analysis 

Additional research questions were explored in the 2014-15 school year, with findings intended to be 
used formatively by the Intervention Team. Findings are provided below, by research question. 

• Was participation in the summer program consistent with the program's goals? Specifically, 
were any students who attended the summer program not enrolled in a Treatment teacher's 
classroom during the academic year? Were any students who attended the summer session 
enrolled in a Control teacher's classroom during the academic year? And, overall, how many 
students participated in the summer program? 

Only grade 7 students assigned to grade 8 Treatment teachers were eligible for the summer program, 
but each district determined which subset of Treatment students would be invited to the summer 
program. Two students, however, who were originally assigned to a classroom taught by a Treatment 
teacher for grade 8 Algebra I and attended the summer program, subsequently experienced schedule 
changes and thus ended up in a classroom taught by an Algebra I teacher who was not participating in 
the study. These students were not included in any analyses. No student who attended the summer 
program was assigned to a Control teacher during the 2012-13 school year. 

• What can we conclude about how implementation affected outcomes? Specifically, was there a 
relationship between the degree to which participating teachers implemented the intervention 
as intended and student-level outcomes? 

WestEd researchers completed a separate study to specifically address this important question. That 
effort culminated in a paper presented at the 2015 annual conference of the American Educational 
Research Association. This paper, entitled Interpreting Intervention Impact through the Lens of 
Implementation Fidelity: Finding from a Federally Funded Evaluation, is provided in Appendix C. 
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Discussion 

This paper presents findings from a federally funded evaluation of an i3 development grant. It describes 
the methods used to study an intervention designed to improve the academic achievement of all 
students in a "gatekeeper" course that is integrally linked to future success in STEM career pathways. 
Algebra I is an entry-level course that is a prerequisite for placement in more advanced courses; it also 
provides the foundation for learning in subsequent courses in the sciences and other content areas. 
Students who do not successfully complete this course are restricted in terms of future academic 
opportunities and are more likely to experience educational marginalization than their higher-achieving 
peers (Adelman, 2006; Gamoran & Hannigan, 2000; Institute of Education Sciences, 2007; Simard, 2009; 
Stoelinga & Lynn, 2013; Wimberly & Noeth, 2005). 

WestEd evaluators found no statistically significant differences in the outcomes of interest, specifically 
between the end-of-grade state test scores for students taught by teachers assigned to the Treatment 
group and those taught by teachers assigned to the Control group. Statistically non-significant results 
are oftentimes the most difficult to interpret, leaving researchers to conjecture about the possible 
explanations for such outcomes. 10 A systematic analysis of implementation, however, can lead 
researchers toward an evidence-based hypothesis that may be a catalyst for future research and 
practice. 

In this study, it was informative to open up the "black box" of implementation so that the underlying 
factors that might have affected the outcomes could be examined (Donaldson & Lipsey, 2006). 
Consideration of implementation information reported by participating teachers provided contextual 
information about the ways in which this intervention was implemented. It appears that grade 8 Algebra 
I teachers face many competing priorities, and allocating the amount of time expected in order to 
implement the intervention was a challenge for many participants. As shown in Figure 1 (pg. 11), 
treatment teachers started with high levels of implementation for each unit, but implementation levels 
dropped toward the ends of units, especially in the second and third units. 

Comments from teachers suggested that the district curriculum was a main priority, as was preparing for 
the state test at the end of the year. Many teachers did not end up spending enough time delivering the 
intervention units or participating in coaching to reach the fidelity thresholds. Representative comments 
from teachers included the following: 

"Students liked the lesson. I just didn't have time to finish." 

"Trying to finish before CST [the California Standards Test] was a challenge." 

"Once again, we had to stop the unit due to CST preparation." 

In addition, teachers who did not implement every unit with fidelity nevertheless had high levels of 
implementation for many lessons. Indeed, the time spent on some lessons by these teachers was 
notably higher than that of their peers. However, levels of implementation for other lessons were 
notably lower. In contrast, teachers who achieved fidelity for all three units had much less volatility in 


10 Such explanations may include, e.g., insufficient power to detect a true effect; an intervention that was not 
implemented with fidelity to the prescribed methods and procedures; or an ineffective intervention. 
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implementation. With the exception of the final lesson in the final unit, the median implementation 
time for each lesson was very close to the time expected by the developers. 

When discussing these patterns with the curriculum developers, WestEd evaluators learned that the last 
one or two lessons in each unit were designed to integrate the concepts discovered through the project- 
based learning with the mathematical concepts delivered through direct instruction within the units. 
Implementing the project-based curriculum without scaffolding that provides appropriate context and 
integration with mathematical processes may have been a severe detriment to the curriculum. Teachers 
spent time with engaging projects (and away from standard curricular instruction aligned with CST 
standards), but many did not follow through to make critical connections essential to the developers' 
intended learning process. 

It is also enlightening to consider the implications of the findings that suggest that students with low- 
fidelity Treatment teachers scored significantly lower on the state test in Algebra I than students taught 
by high-fidelity Treatment teachers. While this is a correlational, not causal, relationship, it nonetheless 
provides instructive feedback to developers about the importance of strict adherence to standardization 
during an RCT, with options for individual choice in which elements of the units to use introduced at the 
conclusion of the RCT. These findings also serve as a catalyst for further investigation into possible links 
between teachers' background characteristics (e.g., number of years of teaching experience, advanced 
coursework in mathematics) and their levels of implementation. 
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LOGIC MODEL: SLOPE (DEV11) v. 13 


INPUTS for Teachers Randomly 


IMPLEMENTATION of Classroom 


MEDIATORS 


STUDENT-LEVEL 

Assigned to Treatment Condition 


Intervention (Prior to and During Grade 8) 


(Underlying Processes) 


OUTCOMES 


Algebra I 

I. Curricular Materials Provided 

• CAMP & Algebra I Curricula + 

• College Awareness Curriculum + 


II. Professional Development 

• Year 1: ConnectEd trains teachers to use 
math curriculum during four half-day 
online sessions and RTF conducts 1 full- 
day training on use of college awareness 
curriculum activities + 

• Year 2: ConnectEd conducts online 
refresher training for one hour quarterly 
(before each unit) and RTF conducts 1 
half-day refresher training + 


III. Coaching 

• Years 1 and 2: CSUN provides online math 
content coaching for 1-3 hours per week + 


Algebra I 

All grade 8 Algebra I students assigned to 
treatment teacher attend year-long Algebra I 
course taught by trained treatment teacher 
IV-VI: 3 ConnectEd Algebra I drop-in units 
(between 7 and 11 class periods each) + 

VII-. 2 RTF College Awareness Curriculum 
activities per Algebra I unit (15 min per 
activity) + 


C.A.M.P. 

VIII. Curricular Materials Provided 

• CAMP & Algebra I Curricula + 

• College Awareness Curriculum + 


IX. Professional Development 

• Year 1: ConnectEd & RTF train teachers to 
use math & college-awareness curriculum 
during 5 full-day meetings + 

• Year 2: ConnectEd & RTF conduct 
refresher training during 2 full-day 
meetings + 


X. Coaching 

• Years 1 and 2: CSUN provides online math 
content coaching for 1 hour per week + 


► 


C.A.M.P. 

All students assigned to treatment teachers are 
invited to attend 4-week summer program 
before grade 8. If space is limited, schools give 
first priority to those students most likely to 
benefit from additional academic support prior 
to Algebra I 

XI-XIII: 3 ConnectEd Pre-Algebra units (Mon- 
Thur) t 

XIV: RTF College Awareness Curriculum (Fri) + 


Teachers' attitude 
and motivation to 
teach improve 

Teachers' 
instructional 
practices improve 

Teachers' content 
knowledge 
improves 


Students benefit 
through 
increased 
engagement in 
learning process 


Improved 
student learning, 
as measured by 
performance on 
state test in 
grade 8 
mathematics 


More students 
enroll in higher- 
level math 
classes and 
pursue STEM 
pathways in high 
school and 
beyond 


LEGEND 



Not measured by i3 grant 

t 

Part of implementation study 

(l-XIV) 

Components of implementation 

measure 
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UUestEd # 


Teacher Background 

Survey 

STEM Learning Opportunities Providing Equity 


Spring 2012 


This survey is being administered by WestEd as part of the STEM Learning Opportunities Providing 
Equity project. This project is funded by the U. S. Department of Education. Several questions in this 
survey are from the National Assessment of Educational Progress 2009 Teacher Background 
Questionnaire for Grade 8 Mathematics. 
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STEM Learning Opportunities Providing Equity 
Teacher Background Survey (RCT) 


Dear Teacher, 

Thank you in advance for completing this survey! This purpose of this survey is to gather your 
demographic information and background information in relation to your role as a mathematics 
teacher. All data you provide will remain confidential and will be used solely for this research study. 
No data you provide will be shared with your school principal or other school/district 
administrators or staff. If you have any questions about this survey, please contact Carol Whang at 
(415] 615-3346 or cwhang@wested.org . 


1. Name: Please provide your first name, middle initial, and last name below. 


First 

M.I. 

Last 





2. Contact Information: Please provide your email address and phone number. We will only 
use this information to contact you if we have questions about any information you submit 
for this study. 


Email address 

Phone Number 


( ) 


3. School Mailing Address: Please provide your school mailing address. We will use this 

information to send study materials directly to you (e.g., for data collection over the course 



4. Gender 

□Female I iMale 

5. Are you Hispanic or Latino? (Select one or more boxes.] 

□No, I am not Hispanic or Latino. 

□Yes, I am Mexican, Mexican American, or Chicano. 

□Yes, I am Puerto Rican or Puerto Rican American. 

□Yes, I am Cuban or Cuban American. 

□Yes, I am from some other Hispanic or Latino background. 


Please go on-> 
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STEM Learning Opportunities Providing Equity 
Teacher Background Survey (RCT) 


6. Which of the following best describes you? (Select one or more boxes.] 

I I White 

I I Black or African American 

I I Asian 

I I American Indian or Alaska Native 

I I Native Hawaiian or other Pacific Islander 

7. Counting this year, how many years have you worked as an elementary or secondary 
teacher? Include any full-time teaching assignments, part-time teaching assignments, and 
long-term substitute assignments, but not student teaching. If less than 4 months total 
experience, enter "00." 


YEARS 


8. Counting this year, how many years have you taught mathematics in grades 6 through 12? 
Include any full-time teaching assignments, part-time teaching assignments, and long-term 
substitute assignments, but not student teaching. If less than 4 months total experience, 
enter "00." 


YEARS 


9. Did you enter teaching through an alternative certification program? (An alternative 
program is a program that was designed to expedite the transition of non-teachers to a 
teaching career, for example, a state, district, or university alternative certification 
program.] 

□Yes DNo 

10. What type of teaching certificate do you hold in the state where you currently teach? 

□Regular or standard state certificate or advanced professional certificate -> Skip to 
Question 12. 

I I Certificate issued after satisfying all requirements except the completion of a 
probationary period ->Go to Question 11. 

□Certificate that requires some additional coursework, student teaching or passage of 
a test before regular certification can be obtained ->Go to Question 11. 

I I Certificate issued to persons who must complete a certification program in order to 
continue teaching -> Go to Question 11. 

I l l do not hold any of the above certifications in the state where I currently teach -> Go 
to Question 11. 


Please go on-> 
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STEM Learning Opportunities Providing Equity 
Teacher Background Survey (RCT) 


11. Do you hold a currently valid regular or standard certification from a state other than the 
one in which you are currently teaching? 

□Yes DNo 

12. This school year, are you a Highly Qualified Teacher (HQT] according to your state's 
requirements? (Generally, to be Highly Qualified, teachers must meet requirements related 
to [1] a bachelor's degree, [2] full state certification, and [3] demonstrate competency in the 
subject area(s] taught. The HQT requirement is a provision under the No Child Left Behind 
(NCLB] Act.] 

□Yes 

I l l meet my state's requirements for a Highly Qualified Teacher in at least one subject 
that I teach. 

□No 

13. Are you certified by the National Board for Professional Teaching Standards in at least one 
content area? (The National Board of Professional Teaching Standards is a 
nongovernmental organization that administers National Board certification, a voluntary 
national assessment program that certifies teachers who meet high professional standards. 
In order to gain certification, the candidate must at least complete a portfolio of classroom 
practice and pass one or more tests of content knowledge.] 

□Yes, I am fully certified by the National Board for Professional Teaching Standards. 

I l l am working toward my National Board certification. 

□No 

14. What is the highest level of education you have completed? (please select one] 

□High school diploma 

□Associate's degree/vocational certification 

□Bachelor's degree 

□Master's degree 

□Education specialist's or professional diploma based on at least one year's work past 
master's degree 

□Doctorate 

□Professional degree (e.g., M.D., LL.B., J.D., D.D.S.] 


Please go on-> 
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STEM Learning Opportunities Providing Equity 
Teacher Background Survey (RCT) 


15. Did you have a major, minor, or special emphasis in any of the following subjects as part of 
your undergraduate coursework? (Select one box for each line.] 



Yes, a major 

Yes, a minor or 
special emphasis 

No 

a. Mathematics education 

□ 

□ 

□ 

b. Mathematics 

□ 

□ 

□ 

c. Other mathematics-related 

□ 

□ 

□ 

subject such as statistics 

d. Education (including secondary 
education] 

□ 

□ 

□ 


16. Did you have a major, minor, or special emphasis in any of the following subjects as part of 
your graduate coursework? (Select one box for each line.] 



Yes, a major 

Yes, a minor or 
special emphasis 

No 

a. Mathematics education 

□ 

□ 

n 

b. Mathematics 

□ 

□ 

□ 

c. Other mathematics-related 

□ 

□ 

□ 

subject such as statistics 

d. Education (including secondary 
education] 

□ 

□ 

□ 


17. Date this survey was completed: 


Month 

Day 

Year 



2012 


Thank you very much for completing this survey! 

Please place your completed survey in the business reply envelope 
(provided) and place the envelope in the mail (no postage is 
necessary). We appreciate you returning this survey as soon as you 

complete it. 
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Introduction 

This research paper describes the methods and preliminary findings from a five-year evaluation of a 
promising middle-school mathematics intervention funded through an Investing in Innovation (i3) 
development grant from the U.S. Department of Education. The objectives of this federally funded study 
are (1) to examine the impact of an intervention aimed at increasing the academic achievement of 
students in a gatekeeper course, 1 Algebra I, as measured by students' end-of-year state test scores in 
mathematics; and (2) to better understand the relationship between intervention impact and 
implementation fidelity, as measured by teachers' levels of compliance with the study protocol. The goal 
of this paper is to suggest a process for meaningfully interpreting findings from a randomized controlled 
trial (RCT), using contextual information about implementation fidelity that has been systematically 
collected during the course of an evaluation. 

Theoretical Framework 

In theory-based evaluations, the systematic examination of implementation of fidelity allows 
experimental-design researchers to open up the "black box," i.e., to better understand the causal 
processes underlying the outcomes that emerge (Donaldson & Lipsey, 2006). Specifically, a fidelity of 
implementation index describes the degree to which key components of the intervention are delivered 
with integrity, through adherence to the developers' intent and/or in keeping with the original program 
design (O'Donnell, 2008). Contextual information that compares actual implementation processes and 
practices to the ideal practices can provide valuable insight into possible explanations for statistically 
significant or non-significant findings (Century, Rudnick, & Freeman, 2010; Hulleman & Cordray, 2009; 
Mowbray, Holter, Teague, & Bybee, 2003; National Research Council, 2004). For evaluations focused on 
measuring the effectiveness of innovative curricular tools, this information may be particularly useful, as 
results can be used formatively by intervention developers to refine processes or procedures prior to 
scale-up (O'Donnell, 2008). 

This study uses an a priori intervention model, in which expectations about implementation are 
specified by the developers and documented by the evaluator prior to data collection (Hulleman & 
Cordray, 2009). These expectations are linked to the intervention's theoretical foundation (theory of 
action or logic model) and become the standard ("threshold") for fidelity for all study participants. The 
logic model for the intervention studied in this evaluation is provided as a figure in Appendix A. As 
shown in that figure, the inputs include curricular materials, training, and professional coaching, which 
support delivery of the instructional units at the classroom level during the summer prior to grade 8 and 
in grade 8. The student-level outcome of interest is performance on the state's standardized test for 
grade 8 in Algebra I, administered annually in late spring. 


1 A gatekeeper course is one that is viewed as of particular importance for success in nearly all career pathways 
associated with gainful employment. See the Implications and Discussion section of this report for additional 
information. 
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Methods 

The intervention evaluated in this study consists of three main components: (1) project-based, drop-in 
instructional units that are strategically incorporated into the existing locally determined Algebra I 
curriculum; (2) project-based Pre-Algebra curriculum units delivered in an optional summer program 
("C.A.M.P."); and (3) professional coaching for participating grade 8 Algebra I teachers. Each component 
is described in greater detail in the following text. 

• The Algebra I instructional units consisted of three project-based science, technology, 
engineering, and mathematics (STEM)-oriented academic units designed to be taught at various 
points during the 2012-13 school year and integrated into the district-selected mathematics 
curriculum. The topics for the units were Puzzle Cube, Air Traffic Control, and Catapult Game. 
Each required the students to work in small teams to apply mathematical concepts to design, 
create, or build a structure or tool. Overall, these three units were expected to take 
approximately forty class periods, with some flexibility in instructional time in order to 
accommodate each district's pre-existing guidelines for timing and sequence of instruction. Only 
teachers assigned to the treatment condition received these curricular materials. All students 
who were enrolled in Algebra I classrooms taught by teachers assigned to the treatment 
condition were exposed to the locally determined curriculum as well as to the intervention 
curriculum. Students taught by control teachers or teachers not participating in the study were 
exposed only to the district-determined curriculum and had no access to the intervention 
curricular materials. 

• Summer C.A.M.P. was a four-week program consisting of three project-based, STEM-oriented 
academic units that required creative problem solving and reinforced mathematics concepts 
and skills needed for success in Algebra I. For example, one unit required students to model and 
build wind turbines that could generate a given amount of power. This unit reviewed the 
concepts of fractions, angles, and constraints. Summer C.A.M.P. was designed for students who 
had not reached proficient status on the grade 7 state test in mathematics in spring 2012; 
however, in a few districts, administrators were not able to fill all program slots with low- 
performing students, so some students who had reached proficient status were allowed to 
attend Summer C.A.M.P. In all districts, student participation was optional. Students assigned to 
classrooms taught by control teachers or teachers not participating in the study were not 
eligible to participate in Summer C.A.M.P. 

• Intervention teacher attendance at professional development and coaching sessions was 
mandatory. Sessions held in the spring, summer, and fall of 2012 focused on implementing the 
instructional units as intended and reinforcing effective instructional strategies. Through the 
2012-13 school year, weekly coaching and collaborative meetings were provided, to strengthen 
teacher understanding of the mathematics content and increase the likelihood of consistent 
implementation across teachers. Control teachers did not receive any coaching or professional 
development from i3 researchers. 

All evaluation activities were regulated by an Institutional Review Board, and the evaluation was 
conducted with oversight from the i3 national evaluation team led by Abt Associates. This team 
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reviewed and approved the design, methodology, and planned analyses for the impact and 
implementation studies. WestEd researchers had conference calls monthly with a representative of this 
team to monitor study progress and discuss emerging challenges. The intent of this oversight was to 
ensure that all i3 evaluations were conducted in compliance with the What Works Clearinghouse 
principles for rigorous scientific research. 

The intervention was piloted in year 2 of the grant (2011-12 school year), followed by a two-year RCT in 
grant years 3 (2012-13 school year) and 4 (2013-14 school year). Data collected in the RCT years were 
focused on impact and exploratory analyses, respectively. Findings from the first year of the RCT— the 
impact study and associated implementation data— are reported in this paper. Each is described in 
greater detail in the following sections. 

A. Impact study: RCT year 1, grant year 3 (2012-13 school year). 

For this study, 70 grade 8 Algebra I teachers were recruited from 15 school districts across California. 
Collectively, these districts are representative of the state in terms of geography, size, demographics, 
and socioeconomic status. Randomization into treatment and control status occurred at the teacher 
level in spring 2012; half of the teachers were randomly assigned to the treatment condition and half to 
the control condition. Students were assigned to classrooms without knowledge of teachers' group 
membership, using each district's routine placement policies. 

Research questions and associated analyses are presented in Table 1. The contrast of interest was 
performance on a standardized Algebra I test by students assigned to Algebra I classrooms taught by 
treatment teachers compared to performance by students assigned to Algebra I classrooms taught by 
control teachers. 

Table 1 

Research Questions and Planned Analyses for Impact Study (RCT Year 1) 


Research Question 

Analysis 

Contrast 1: On average, for participating students whose performance was 
at or above proficient on the grade 7 state test in mathematics administered 
in 2012, does performance on the state's grade 8 test in Algebra 1 differ 
between students who are exposed to the grade 8 Algebra 1 intervention and 
students who are not exposed to the grade 8 Algebra 1 intervention? 

A 2-level hierarchical linear model 
(HLM) that examines the impact of 
project-based drop-in units on 
grade 8 test scores collected in 
spring 2013 

Contrast 2: On average, for participating students whose performance was 
below proficient on the grade 7 state test in mathematics administered in 
2012, does performance on the state's grade 8 test in Algebra 1 differ 
between students who are exposed to the grade 8 Algebra 1 intervention 
plus C.A.M.P. and students who are not exposed to these two components 
of the intervention? 

A 2-level HLM that examines the 
impact of project-based drop-in 
units and Summer C.A.M.P. on 
grade 8 test scores collected in 
spring 2013 

Contrast 3: On average, for all students participating in this study with a 
score from the grade 7 state tests in mathematics administered in 2012, 
does performance on the state's grade 8 test in Algebra 1 differ for students 
who are exposed to any intervention components and students who are not 
exposed to any intervention components? 

A 2-level hierarchical linear model 
(HLM) that examines the impact of 
the intervention on grade 8 test 
scores collected in spring 2013 
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B. Implementation study. 

The focus of the implementation study was on better understanding the ways in which participating 
teachers assigned to the treatment condition carried out the required components of the intervention, 
and the choices they made during the course of the study. Such information, collected from each 
participating teacher and then aggregated up to the program level, may shed light on how fidelity of 
implementation affected the student learning outcomes as specified in the logic model. This information 
allows us to more meaningfully interpret the impact findings. 

The core components of the intervention included use of the prescribed mathematics curricular 
materials during Algebra I instruction, attendance at training activities, and participation in coaching 
sessions. These components were mandatory, but within each component, teachers could make 
individual choices about certain activities. For example, each drop-in unit was divided into lessons, and 
lessons were subdivided into steps (e.g., worksheets, activities, instructional modules). Teachers were 
not required to use every lesson in a particular unit in order to reach the threshold for fidelity. In 
addition, those teachers who were assigned to the treatment condition and who elected to teach a 
Summer C.A.M.P. session were required to use prescribed mathematics curricular materials during 
instruction and to attend a special training session. 

The method used in this study to evaluate fidelity of implementation included the following steps: 

1. Specification of the intervention logic model (theory of action) that identifies all key 
components of the intervention, the mediating factors through which the intervention is 
implemented, and the outcomes that the intervention is designed to achieve; 

2. Identification of the instruments (e.g., surveys, attendance rosters, and other 
documentation) that will be used to collect information about each teacher's experiences; 

3. Development of a matrix that operationalizes the constructs of interest in terms of the 
observable indicators that are collected and that can be used to evaluate the degree to which 
each component was implemented "with fidelity," or in accordance with the developers' 
intent; and 

4. Calculation of a composite fidelity index for each component (e.g., training, coaching, or 
curriculum unit) for each teacher (teacher-level score) and for the entire sample (program- 
level score). 

This method calls for clear linkage among the intervention logic model, which explains the hypothesized 
cause for the differences in outcomes between the treatment and control conditions, and the selected 
fidelity indices. Importantly, it also calls for the thresholds for fidelity to be determined by the 
intervention developers, with support from the evaluator. This methodology promotes study coherence, 
ensures that findings that emerge from the implementation fidelity analyses are meaningfully associated 
with the findings from impact analyses, and supports the validity of findings that emerge from the 
implementation study. 


2 The techniques described in this section were developed in conjunction with the i3 National Evaluator, Abt 
Associates (Barbara Goodson, Principal Investigator), and its partners. 
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Consistent with this method, we designed a sophisticated fidelity matrix (see Appendix B for an 
example) that is linked to the logic model and that aims to capture meaningful and discernible levels of 
implementation among teachers. 

Data Sources 

Data were collected during year 1 of the RCT in order to study intervention impact and implementation 
fidelity; these data are the focus of this paper. For all participating students (those in Algebra I classes 
taught by teachers assigned to treatment or control conditions), demographic information (e.g., gender, 
race/ethnicity, free/reduced-price lunch status, English learner status, and special education status) and 
test scores were collected during the 2013 end-of-year administration of the California Standards Test 
(CST) in Algebra I. For baseline equivalency testing, evaluators collected the prior year's (grade 7 in 
2012) test scores. WestEd researchers matched data from the two years for each participating student. 

In spring 2011, immediately following randomization, background information was collected from each 
participating treatment and control teacher via an online Teacher Background Survey (see Appendix C). 
In fall 2012, each participating treatment and control teacher also provided a roster listing the names of 
all students in their classrooms at that time. 

Teachers who were assigned to the treatment condition and who elected to teach a Summer C.A.M.P. 
session in summer 2012 recorded information about each unit that they taught in an implementation 
survey. In addition, throughout the 2012-13 school year, teachers assigned to the treatment condition 
(including those who taught Summer C.A.M.P. sessions) who were teaching Algebra I that year 
completed an implementation survey, as they finished each unit of instruction, and provided specific 
information about the instructional activities in which they had participated and time spent in each 
activity. In that survey, treatment teachers were also asked to answer specific questions about their 
experiences with the coaching component of the study. An example of an end-of-unit implementation 
survey is included in Appendix D. 

The study director provided documentation of teacher participation in mandatory training and 
professional development sessions to the evaluation team. 

A summary of data sources is provided in Table 2. 
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Table 2 

Data Sources for 2012-13 RCT and Implementation Studies 


Study 

Student or Teacher Population 

Student-Level Data 

Source 

Teacher-Level Data 

Source 

RCT 

All grade 8 students in participating 
Algebra 1 classrooms (treatment and 
control) 

State test (CST) in 
Algebra 1 in spring 

2013 


Implementation 

Study 

All participating treatment and 
control teachers 


Teacher Background Survey 
Class roster 

Implementation 

Study 

All grade 8 Algebra 1 teachers 
assigned to the treatment condition 
who elected to teach C.A.M.P. 


C.A.M.P Implementation 
Survey (by unit) 

Implementation 

Study 

All grade 8 Algebra 1 teachers 
assigned to the treatment condition 


Algebra 1 Implementation 
Survey (by unit) 
Documentation from study 
director verifying 
attendance at training 


Findings 

The final analytic sample for the 2012-13 cohort included 1,384 students assigned to 28 treatment 
teachers and 1,088 students assigned to 27 control teachers. 3 Table 3 presents descriptive information 
about the teacher participants. 

Table 3 

Descriptive Information About Teacher Cohort, 2012-13 School Year 



n 

Percentage 

Male 

Average Years 
Mathematics 
Teaching Experience 

Percentage with 
Degree Beyond 
Bachelor's 

Treatment 

28 

39% 

10.86 

32% 

Control 

27 

11% 

8.52 

30% 

Total 

55 





Students who were included in the 2012-13 cohort met the following criteria: (1) were enrolled in an 
Algebra I class taught by a participating treatment or control teacher; (2) were classified as a grade 8 
student by the district; (3) had a valid test score for grade 8 Algebra I; and (4) did not opt out of data 
collection (most districts) or had parent permission to participate in data collection (one district). 4 All 
other students were included for instruction but excluded for impact analyses. 

Preliminary findings from the analyses are provided in Table 4. 


3 Eight teachers were not eligible to participate in the study, as they were not assigned to teach grade 8 Algebra I in 
fall 2012. The remaining attriters, from the original cohort of 70 teachers, exited the study for personal reasons. 

4 The majority of schools/districts had an "opt-out" policy. This means that a student's guardian had to return a 
waiver in order for the student to be excluded from the study. However, one district, the largest district in the 
study, insisted on an "opt-in" policy. This means that a student's guardian had to return a waiver in order for the 
student to be included in the study. 
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Table 4 

Research Questions and Planned Analyses for Impact Study (RCT Year 1) 


Research Question 

n 

Mean Test Score 

T/C Mean Difference 
(S.D.) 

p-Value 

Contrast 1 

T = 673 

T = 351 

-6.01 

.210 


C = 519 

C = 357 

(4.798) 


Contrast 2 

T= 149 

T = 282 

-5.73 

.086 


C = 452 

C = 288 

(3.336) 


Contrast 3 

T= 1,384 

T = 320 

-5.38 

.174 


C = 1,088 

C = 325 

(3.953) 



*significant at .05 level 

As shown in Table 4, no contrast showed a statistically significant difference at the .05 level. Students 
who were assigned to classrooms taught by treatment teachers did not perform differently in relation to 
those assigned to classrooms taught by control teachers. 


Implementation study. 

In Appendix B (one sample unit: Unit 1), we describe the levels of implementation that the developers, 
in consultation with the research team, set as the thresholds for fidelity. Appendix B also includes 
teacher-reported information about the number of lessons used, the time spent on each lesson, the 
steps used within each lesson, and the estimated amount of effort needed for each lesson. The indexing 
scheme included in Appendix B provides an overall assessment of an individual teacher's fidelity to the 
implementation model. 


Preliminary findings from the implementation study for Algebra I teachers are presented in Table 5. 
Fidelity findings are presented for the coaching component and for each Algebra I drop-in unit. 5 
Thresholds for program-level fidelity (80% or more of teachers meeting individual thresholds for each 
component) are also indicated. 


Table 5 

Preliminary Implementation Study Findings, by Component 


Component 


Description 


Number of 
Teachers 
Meeting Fidelity 
Threshold 


Percent of 
Teachers 
Meeting Fidelity 
Threshold 


Program- 

Level 

Fidelity 


Coaching 

Teacher participates in an average of 
one or more hours of coaching and 

16/28 

57% 

No 


collaboration sessions for each unit. 





5 Implementation fidelity was assessed for every part of the intervention, including the distribution of curricular 
materials, participation in professional development seminars, and implementation of College Awareness 
Curriculum activities. In addition, WestEd researchers conducted a full implementation study for the Summer 
C.A.M.P. program. Only the results for coaching and collaboration time and direct implementation of the curricular 
units are reported and discussed in this paper. 
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Unit 1: 

Puzzle Cube 

Teacher implements Unit 1 of 
intervention curriculum. 

25/28 

89% 

Yes 

Unit 2: 

Air Traffic Control 

Teacher implements Unit 2 of 
intervention curriculum. 

18/28 

64% 

No 

Unit 3: 

Catapult Game 

Teacher implements Unit 3 of 
intervention curriculum. 

21/28 

75% 

No 


As shown in Table 5, in terms of the coaching component, while every teacher participated in some level 
of coaching, only 16 of the 28 Algebra I teachers participated regularly enough and for enough time to 
reach the threshold for fidelity. The implementation expectations for each of the core elements of this 
intervention, the three drop-in units, are reported separately. The percentages of teachers reaching the 
fidelity threshold ranged from 64% for Unit 2 to 89% for Unit 1. Teacher level data showed that only 16 
of the 28 treatment teachers met the threshold for fidelity for all three units. 

The marks corresponding to the graph's left axis in Figure 1 show the average percentages of expected 
time that teachers spent on each lesson within the Algebra I drop-in units. 


Figure 1. 

Percentages of expected time spent on each lesson. 



As an example, the curriculum developers planned for the first lesson of the first unit ("Puzzle 1") to 
take seventy-five minutes of class time. If the average implementation time for teachers who reported 
using this lesson at all were exactly seventy-five minutes, the mark would be exactly at 1. As shown in 
Figure 1, the average implementation time was 101% of the expected time. If the average were below 
100%, the mark would fall below 1, and vice versa for an average higher than the anticipated rate. 
Additionally, the bars corresponding to the graph's right axis show the numbers of teachers who did not 
implement a given lesson at all. A taller bar indicates that more teachers did not implement any steps in 
the given lesson. As shown in Figure 1, treatment teachers appear to start with high levels of 
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implementation for each unit, but drop-offs in implementation levels are increasingly evident toward 
the ends of units, especially for the second and third units. 

Figures 2a and 2b show the medians and interquartile ranges of the data that are shown in Figure 1. The 
data in these figures are divided into two subsamples of treatment teachers: treatment teachers who 
implemented all of the units with fidelity (Figure 2a) and those who did not implement one or more of 
the units with fidelity (Figure 2b). 

Figure 2a. 

Teachers implementing all curriculum units with fidelity. 

Percentage of Expected Time Used for Each 
Lesson by "High-Implementing" Teachers 
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Figure 2b. 

Teachers who did not implement one or more curriculum units with fidelity. 


Percentage of Expected Time Used for Each 
Lesson by "Low-Implementing" Teachers 



— Median 

- First Quartile 

- Third Quartile 


As shown in Figure 2a, teachers who implemented all three units with fidelity generally had a median 
amount of time spent on each lesson at or just above the expectation. In contrast, as shown in Figure 
2b, teachers who did not implement every unit with fidelity had much more variation in the time spent 
on implementation across lessons. 


Importantly, among the 28 treatment teachers, 16 appear to have completed all three required units 
with a high level of fidelity, while 12 completed the units with a mix of high and low levels of fidelity. To 
explore the effect of these different rates of implementation in greater depth, we conducted an impact 
analysis that was similar to Contrast 3 (all treatment and control students included), examining three 
teacher groups: treatment teachers with a high level of fidelity on all three units (n = 16) vs. other 
treatment teachers (n = 12) vs. control teachers (n = 24). We chose to focus our investigation on the 
curricular units because they are the core components of the intervention. The Contrast 3 sample was 
used so that there would be more students in each of the two newly constructed treatment groups, and 
thus more statistical power to detect possible differences. 6 

Table 6 summarizes the findings from this analysis. 


6 It should be noted, however, that the study was not powered to detect such group differences. Non-significant 
differences could be primarily due to an insufficient number of teachers in comparison groups. 
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Table 6 

Impact Analysis on Student Algebra I Scores, by Three Teacher Groups 

Adjusted Mean 

Mean Difference 


Group Membership 


P- 

Value 


Effect 

Size 


Treatment (low fidelity) 

581 

311.62 




Treatment (high fidelity) 

803 

324.65 




Control 

1,088 

324.62 




Diff: T-low and C 



-13.00 

0.039* 

-0.21 

Diff: T-high and C 



0.03 

0.996 

0.00 

Diff: T-low and T-high 



-13.03 

0.048* 

-0.21 


*significant at .05 level 


As shown in Table 6, students with high-fidelity treatment teachers did not perform differently from 
control students on the Algebra I assessment. Students with low-fidelity treatment teachers, however, 
scored significantly lower than control students. Similarly, students with low-fidelity treatment teachers 
also scored significantly lower than students with high-fidelity treatment teachers. 


Implications and Discussion 

This paper presents findings from a federally funded evaluation of an intervention designed to improve 
the academic achievement of all students in a gatekeeper course that is integrally linked to future 
success in STEM career pathways. Algebra I is an entry-level course that is a prerequisite for placement 
in more advanced courses; it also provides the foundation for learning in subsequent courses in the 
sciences and other content areas. Students who do not successfully complete this course are restricted 
in terms of future academic opportunities and are more likely to experience educational marginalization 
than their higher-achieving peers (Adelman, 2006; Gamoran & Hannigan, 2000; Institute of Education 
Sciences, 2007; Simard, 2009; Stoelinga & Lynn, 2013; Wimberly & Noeth, 2005). 

In this study, no statistically significant differences in the outcomes of interest emerged between 
students taught by teachers assigned to the treatment condition and those taught by teachers assigned 
to the control condition. Statistically non-significant results are oftentimes the most difficult to 
interpret, leaving researchers to conjecture about the possible explanations for such outcomes. 7 A 
systematic analysis of implementation, however, can lead researchers toward an evidence-based 
hypothesis that may be a catalyst for future research and practice. In this study, it was informative to 
open up the "black box" of implementation so that the underlying factors that might have affected the 
outcomes could be examined (Donaldson & Lipsey, 2006). 

In this study, consideration of implementation information reported by participating teachers provided 
contextual information about the way in which this intervention was implemented. It appears that grade 
8 Algebra I teachers face many competing priorities, and allocating the amount of time expected in 
order to implement the intervention was a challenge for many participants. Comments from teachers 
suggest that the district curriculum was a main priority, as was preparing for the state test at the end of 

7 Such explanations may include, e.g., insufficient power to detect a true effect; an intervention that was not 
implemented with fidelity to the prescribed methods and procedures; or an ineffective intervention. 
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the year. Many teachers did not end up spending enough time delivering the intervention units or 
participating in coaching to reach the fidelity thresholds. As shown in Figure 1, treatment teachers 
started with high levels of implementation for each unit, but implementation levels dropped toward the 
ends of units, especially in the second and third units. Representative comments from teachers included 
the following: 

"Students liked the lesson. I just didn't have time to finish." 

"Trying to finish before CST [the California Standards Test] was a challenge." 

"Once again, we had to stop the unit due to CST preparation." 

Figures 2a and 2b provide a more detailed picture of the trends shown in Figure 1. Teachers who did not 
implement every unit with fidelity nevertheless had high levels of implementation for many lessons. 
Indeed, the time spent on some lessons by these teachers was notably higher than that of their peers. 
However, levels of implementation for other lessons were notably lower. In contrast, teachers who 
achieved fidelity for all three units had much less volatility in implementation. With the exception of the 
final lesson in the final unit, the median implementation time for each lesson was very close to the time 
expected by the developers. 

When discussing these patterns with the curriculum developers, we discovered that the last one or two 
lessons in each unit were designed to integrate the concepts discovered through the project-based 
learning with the mathematical concepts delivered through direct instruction within the units. 
Implementing the project-based curriculum without scaffolding that provides appropriate context and 
integration with mathematical processes may have been a severe detriment to the curriculum. Teachers 
spent time with engaging projects (and away from standard curricular instruction aligned with CST 
standards), but many did not follow through to make critical connections essential to the developers' 
intended learning process. 

It is also enlightening to consider the implications of the data presented in Table 6. Findings suggest that 
students with low-fidelity treatment teachers scored significantly lower on the state test in Algebra I 
than students taught by high-fidelity treatment teachers. While this is a correlational, not causal, 
relationship, it nonetheless provides instructive feedback to developers about the importance of strict 
adherence to standardization during an RCT, with options for individual choice in which elements of the 
units to use introduced at the conclusion of the RCT. These findings also serve as a catalyst for further 
investigation into possible links between teachers' background characteristics (e.g., number of years of 
teaching experience, advanced coursework in mathematics) and their levels of implementation. 
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Appendix A. i3 Intervention Logic Model 


INPUTS for Teachers Randomly 


IMPLEMENTATION of Classroom 


MEDIATORS 


STUDENT-LEVEL 

Assigned to Treatment Condition 


Intervention (Prior to and During Grade 8) 


(Underlying Processes) 


OUTCOMES 


Curricular Materials (I) 

• C.A.M.P. & Algebra I curricula + 

• College Awareness Curriculum + 

Algebra I PD (II) 

• Year 1: ConnectEd trains teachers to use 
math curriculum during four half-day 
online sessions and RTF conducts 1 full- 
day training on use of college awareness 
curriculum activities + 

• Year 2: ConnectEd conducts online 
refresher training for one hour quarterly 
(before each unit) and RTF conducts 

1 half-day refresher training + 

Coaching (III) 

• Years 1 and 2: CSUN provides online math 
content coaching for 1-3 hours per week 
t 


>• 


Algebra I (IV) 

All grade 8 Algebra I students assigned to 
treatment teacher attend year-long Algebra I 
course taught by trained treatment teacher 

• 3 ConnectEd Algebra I drop-in units 
(between 7 and 11 class periods each) + 

• 6 RTF College Awareness Curriculum 
activities (15 min. each) + 


Curricular Materials (V) 

• C.A.M.P. & Algebra I curricula + 

• College Awareness Curriculum + 

CAMP PD (VI) 

• Year 1: ConnectEd and RTF train teachers 
to use math and college awareness 
curriculum during 5 full-day meetings + 

• Year 2: ConnectEd and RTF conduct 
refresher training during 2 full-day 
meetings + 

Coaching (VII) 

• Years 1 and 2: CSUN provides online math 
content coaching for 1 hour per week + 


CAMP (VIII) 

All students assigned to treatment teachers are 
invited to attend 4-week summer program 
before grade 8. If space is limited, schools give 
first priority to those students most likely to 
benefit from additional academic support prior 
to Algebra I. 

• 3 ConnectEd Pre-Algebra units (Mon.-Thur.) 
t 

• RTF College Awareness Curriculum (Fri.) + 


fipact Through Implementation Fidelity 


Students benefit 
through 
increased 
engagement in 
learning process 


Teachers' attitudes 
and motivation to 
teach improve 


Teachers' 
instructional 
practices improve 

Teachers' content 
knowledge 
improves 




Improved 
student learning, 
as measured by 
performance on 
state test in 
grade 8 
mathematics 


More students 
enroll in higher- 
level math 
classes and 
pursue STEM 
pathways in high 
school and 
beyond 



LEGEND 



Not measured by i3 grant 



t 

Part of implementation study 
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2012-2013 Measuring Fidelity of Implementation for Algebra I Drop-in Units: DEV11 (SLOPE) 



Component Description 

Operational Definition 

Data Collection 
Measure 

Possible 

Implementation 

Range 

Expected Teacher- 
level 

Implementation 
with Fidelity / 
Corresponding 
Score 

Teacher-level 
Composite Score 
/ Criterion for 
Adequate 
(implementation 
"with fidelity") 

Program-level 
Criterion for 
Adequate 
(implementation 
"with fidelity") 

Considered 
"with fidelity" 
for Year 1 

Component III. Coaching (online) 

4 

Y 1 Algebra 1 Coaching and 
Collaboration for all treatment 
teachers (online sessions 
scheduled for 1-3 hours/week): 
Unit 1 

Teacher discusses at least 
one issue with coach 
and/or interacts with 
coach and other teachers 

Algebra I 

Implementation Survey 
for Puzzle Cube 
(Questions 19-21), 
and/or coaching log 

0-3 hrs per week 

Participated at least 1 
hour per week / No 
or Yes: 0-1 

0-3 

Adequate: 3 

80% of teachers 
met criterion 

No or Yes 

Components IV-a, IV-b, IV-c. Teacher Implementation of Classroom Intervention (Algebra I Units 1, 2, and 3 Respectively) 


Component: 

3 Algebra I units 

Operational Definition 

Data Collection 
Measure 

Possible 

Implementation 

Range 

Expected Teacher- 
level 

Implementation 
with Fidelity / 
Corresponding 
Score 

Teacher-level 
Composite Score 
/ Criterion for 
Adequate 
(implementation 
"with fidelity") 

Program-level 
Criterion for 
Adequate 
(implementation 
"with fidelity") 

Considered 
"with fidelity" 
for Year 1 

7 

IV-a 

Y 1 Algebra I: Implementation 
of Algebra Drop-in Unit 1 

Teacher implements 
ConnectEd math 
curriculum as trained in 

PD 

Algebra I 

Implementation Survey 
for Puzzle Cube 
(Questions 2-18) 








a. Number of lessons 
used 


a. 0-5 lessons 

a. > 80% of 
implementation 
range (i.e., 4 lessons) 

/ No or Yes: 0-1 

0-16 

Adequate: > 75% 
of possible highest 
composite score; 

i.e., the score for 
implementation 
with fidelity is at 
least 12 

(this indicates that 
the minimal 
number of lessons 
to be taught is 4; 
no way to reach a 

80% of teachers 
meet criterion 

No or Yes 



b. Time spent on each 
lesson 


b. 0-150 minutes 
(varies for each 
lesson) 

b. > 70% of 
implementation 
range for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons) 
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c. Number of steps used 
within each lesson 


c. 0-8 steps (varies 
for each lesson) 
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b. > 70% of score of 12 or 

implementation above if teachers 

range for each taught cover 3 or less 
lesson lessons) 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons) 

d. > 2 for each taught 
lesson / No or Yes: 

0- 1 for each taught 
lesson 

(0-5 across all 

lessons) 

b. > 70% of 

implementation 

range for each taught 

lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons) 

b. > 70% of 
implementation 
range for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons) 

d. > 2 for each taught 
lesson / No or Yes: 

0- 1 for each taught 
lesson 

(0-5 across all 
lessons) 
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Dear Teacher, 

Thank you in advance for completing this survey! This purpose of this survey is to gather your demographic 
information and background information in relation to your role as a mathematics teacher. All data you provide will 
remain confidential and will be used solely for this research study. No data you provide will be shared with your school 
principal or other school/district administrators or staff. If you have any questions about this survey, please contact Dr. 
Carole Gallagher at cgallag@wested.org. 


1. Name: Please provide your first name, middle initial, and last name below. 


First 

M.I. 

Last 





2. Contact Information: Please provide your email address and phone number. We will only use this information to 
contact you if we have questions about any information you submit for this study. 


Email address 

Phone Number 


( ) 


3. School Mailing Address: Please provide your school mailing address. We will use this information to send study 
materials directly to you (e.g., for data collection over the course of this 2-year study]. 



4. Gender 

□Female I iMale 

5. Are you Hispanic or Latino? (Select one or more boxes.] 

□No, I am not Hispanic or Latino. 

□Yes, I am Mexican, Mexican American, or Chicano. 

□Yes, I am Puerto Rican or Puerto Rican American. 

□Yes, I am Cuban or Cuban American. 

□Yes, I am from some other Hispanic or Latino background. 


Please go on-> 


6. Which of the following best describes you? (Select one or more boxes.] 
I I White 

I I Black or African American 
I I Asian 
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| | American Indian or Alaska Native 

I I Native Hawaiian or other Pacific Islander 

7. Counting this year, how many years have you worked as an elementary or secondary teacher? Include any full- 
time teaching assignments, part-time teaching assignments, and long-term substitute assignments, but not 
student teaching. If less than 4 months total experience, enter "00." 

YEARS 

8. Counting this year, how many years have you taught mathematics in grades 6 through 12? Include any full-time 
teaching assignments, part-time teaching assignments, and long-term substitute assignments, but not student 
teaching. If less than 4 months total experience, enter "00." 

YEARS 

9. Did you enter teaching through an alternative certification program? (An alternative program is a program that 
was designed to expedite the transition of non-teachers to a teaching career, for example, a state, district, or 
university alternative certification program.] 

□Yes DNo 

10. What type of teaching certificate do you hold in the state where you currently teach? 

□Regular or standard state certificate or advanced professional certificate -> Skip to Question 12. 

I I Certificate issued after satisfying all requirements except the completion of a probationary period ->Go 
to Question 11. 

□Certificate that requires some additional coursework, student teaching or passage of a test before regular 
certification can be obtained ~^Go to Question 11. 

I I Certificate issued to persons who must complete a certification program in order to continue teaching -> 
Go to Question 11. 

I l l do not hold any of the above certifications in the state where I currently teach Go to Question 11. 


Please go on-> 


11. Do you hold a currently valid regular or standard certification from a state other than the one in which you are 
currently teaching? 

□Yes QNo 

12. This school year, are you a Highly Qualified Teacher [HQT] according to your state's requirements? (Generally, 
to be Highly Qualified, teachers must meet requirements related to (1] a bachelor's degree, (2] full state 
certification, and (3] demonstrate competency in the subject area(s] taught. The HQT requirement is a provision 
under the No Child Left Behind (NCLB] Act.] 

□Yes 

I l l meet my state's requirements for a Highly Qualified Teacher in at least one subject that I teach. 

□No 
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13. Are you certified by the National Board for Professional Teaching Standards in at least one content area? (The 
National Board of Professional Teaching Standards is a nongovernmental organization that administers National 
Board certification, a voluntary national assessment program that certifies teachers who meet high professional 
standards. In order to gain certification, the candidate must at least complete a portfolio of classroom practice 
and pass one or more tests of content knowledge.] 

□Yes, I am fully certified by the National Board for Professional Teaching Standards. 

I l l am working toward my National Board certification. 

□No 

14. What is the highest level of education you have completed? (please select one] 

□High school diploma 

□Associate's degree/vocational certification 

□Bachelor's degree 

□Master's degree 

□Education specialist's or professional diploma based on at least one year's work past master's degree 

□Doctorate 

□Professional degree (e.g., M.D., LL.B., J.D., D.D.S.] 


Please go on-> 


15. Did you have a major, minor, or special emphasis in any of the following subjects as part of your undergraduate 
coursework? (Select one box for each line.] 



Yes, a major 

Yes, a minor or 
special emphasis 

No 

Mathematics education 

□ 

□ 

□ 

Mathematics 

□ 

□ 

□ 

Other mathematics-related 
subject such as statistics 

□ 

□ 

□ 

Education (including 
secondary education) 

□ 

□ 

□ 


16. Did you have a major, minor, or special emphasis in any of the following subjects as part of your graduate 
coursework? (Select one box for each line.] 


a. Mathematics education 


Yes, a major 


□ 


Yes, a minor or 
special emphasis 


□ 
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b. Mathematics 

c. Other mathematics-related 
subject such as statistics 

d. Education (including 
secondary education) 

17. In the past two (2) years, have you participated in any professional development activities specifically 
addressing 8 th grade mathematics? 

□Yes DNo 

18. If you answered Yes to question 17, approximately how many days of professional development addressing 8 th 
grade mathematics have you participated in over the past two (2) years? 

DAYS 

19. Date this survey was completed: 


Month 

Day 

Year 



2012 


Thank you very much for completing this survey! 
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Appendix D. Example Implementation Survey 

UJestEd #. 

Implementation Survey for 
Air Traffic Control 

STEM Learning Opportunities Providing Equity 


This survey is being administered by WestEd as part of the STEM Learning Opportunities Providing Equity 
project. This project is funded by the U. S. Department of Education. 
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STEM Learning Opportunities Providing Equity 

Algebra I Implementation Survey: Air Traffic Control (RCT) 

Dear teacher, 


Thank you in advance for completing this survey! The purpose of this survey is to gather information on the 
implementation of the project-based math curriculum, the related Coaching & Collaboration, and the College 
Awareness Curriculum [CAC], All data you provide will remain confidential and will be used solely for this 
research study. No data you provide will be shared with your school principal or other school/district 
administrators or staff. If you have any questions about this survey, please contact Carol Whang at (415] 615- 
3346 or cwhang@wested.org. 


1. Name: Please provide your first name, middle initial, and last name below. 


First 

M.I. 

Last 





2. Contact Information: Please provide your email address and phone number. We will only use this 
information to contact you if we have questions about any information you submit for this study. 


Email address 

Phone Number 


( ) 


3. Date you started teaching the Air Traffic Control unit (e.g. 9/3/07]: / /. 


The Air Traffic Control unit has 8 lessons. Questions 4-29 pertain to the Air Traffic Control lessons. 


4. Did you use Lesson 1? 


Yes 


□ 


No 


If your answer is No, skip questions 5 and 6, then go to question 7. 


□ 


5. Lesson 1 was anticipated to take 100 minutes 
(2 periods]. Approximately how many minutes 
did you spend on Lesson 1? 

minutes 
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6. Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Please tell us how much attention you gave to each 
step when teaching Lesson 1. 


Did Did and gave this step . . . 

not do on ly a 

little some a lot of 

attention attention attention 


a. 

Introducing the unit and showing videos 

□ 

□ 

□ 

□ 

b. 

Working through Delta 134 problem 

□ 

□ 

□ 

□ 

c. 

Working through Southwest 190 problem 

□ 

□ 

□ 

□ 

d. 

Working through American 722 problem 

□ 

□ 

□ 

□ 

e. 

Discussion of x- and y-intercept 

□ 

□ 

□ 

□ 


7. Did you use Lesson 2? 

Yes 

No 


□ 

□ 


If your answer is No, skip questions 8 and 9, then go to question 10. 


8. Lesson 2 was anticipated to take 75 minutes (1.5 
periods]. Approximately how many minutes did 

you spend on Lesson 2? minutes 


9. Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Please tell us how much attention you gave to each 
step when teaching Lesson 2. 


Did 
not do 


Did and gave this step . . . 


only a 

little some a lot of 

attention attention attention 
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a. Class discussion on rate of change 

□ 

□ 

□ 

□ 

b. Student pairs completing Southwest 190 problem 

□ 

□ 

□ 

□ 

c. Reviewing graphing rate of change homework 

□ 

□ 

□ 

□ 


□ □ 


10 . 

Did you use Lesson 3? 

Yes 

No 


If your answer is No, skip questions 11 and 12, then go to question 13. 


11 . 

Lesson 3 was anticipated to take 100 minutes (2 
periods]. Approximately how many minutes did 


minutes 


you spend on Lesson 3? 



12 . 

Often, it is impossible to do all the steps in a lesson or 

Did 

Did and gave this step . . . 


to give each step as much attention as you would like. 
Please tell us how much attention you gave to each 
step when teaching Lesson 3. 

not do 

only a 

little some a lot of 

attention attention attention 


a. 

Direct instruction on slope 

b. 

Opening class work Delta 134 problem 

c. 

Student pairs completing Delta 134 problem 

d. 

Reviewing calculating slopes homework 


e. Creation and display of slope/rate of change posters 


13. Did you use Lesson 4? 

Yes 

No 


□ 

□ 


If your answer is No, skip questions 14 and 15, then go to question 16. 


14. Lesson 4 was anticipated to take 75 minutes (1.5 
periods]. Approximately how many minutes did 

you spend on Lesson 4? minutes 
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STEM Learning Opportunities Providing Equity 


Algebra I Implementation Survey: Air Traffic Control (RCT) 

Did and gave this step . . . 


15. 

Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Please tell us how much attention you gave to each 
step when teaching Lesson 4. 

Did 

not do 

only a 

little 

attention 

some 

attention 

a lot of 

attention 

a. 

Discussion of constant distance 

□ 

□ 

□ 

□ 

b. 

Student pairs completing American 722 problem 

□ 

□ 

□ 

□ 

c. 

Reviewing parallel lines homework 

□ 

□ 

□ 

□ 



□ 

□ 



16. 

Did you use Lesson 5? 

Yes 

No 




If your answer is No, skip questions 17 and 18, then go to question 19. 

17. Lesson 5 was anticipated to take 75 minutes (1.5 

periods). Approximately how many minutes did minutes 

you spend on Lesson 5? 
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18 . Often, it is impossible to do all the steps in a lesson or Did 

to give each step as much attention as you would like. not do 

Please tell us how much attention you gave to each 
step when teaching Lesson 5. 

a. Introduction to Safe Landings problem 

b. Individual work on Safe Landings problem 

c. Reviewing different solutions to Safe Landings 


Did and gave this step 
only a 

little some 

attention attention 


19 . Did you use Lesson 6? 

Yes 

No 


□ 

□ 


If your answer is No, skip questions 20 and 21, then go to question 22. 

20 . Lesson 6 was anticipated to take 150 minutes (3 
periods]. Approximately how many minutes did 

you spend on Lesson 6? minutes 

21 . Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Did and gave this 

Did onl y a 

not do 

Please tell us how much attention you gave to each little 

step when teaching Lesson 6. attention 




some 

attention 


a lot of 
attention 


a lot of 
attention 
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24. Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Did 

not do 

Did and gave this step . 

only a 

little some 

a lot of 

a. Review of previous in Lesson Introduction 

□ 

□ 


□ 

□ 

b. Discussion on slope and linear equation 
connection 

□ 

□ 


□ 

□ 

c. Reviewing What Is Your Position 

problem/Discussion of calculation of slope from 
two points 

□ 

□ 


□ 

□ 

d. Introduction to Big Picture /two variable 
problem 

□ 

□ 


□ 

□ 

e. Reviewing Big Picture problem 

□ 

□ 


□ 

□ 

f. Student work on graphing equations sheet 
[6.3) 

□ 

□ 


□ 

□ 

g. Reviewing graphing equations sheet (6.3) 

□ 

□ 


□ 

□ 


□ 


□ 



22. Did you use Lesson 7? 

Yes 


No 



If your answer is No, skip questions 23 and 24, then go to question 25. 





23. Lesson 7 was anticipated to take 75 minutes (1.5 
periods). Approximately how many minutes did 



minutes 


you spend on Lesson 7? 

Please tell us how much attention you gave to each 
step when teaching Lesson 7. 






a. Independent work on Which Form Is Best? 

□ 

□ 


□ 

□ 



b. Discussion on slope-intercept vs. standard form 

□ 

□ 


□ 

□ 
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c. 

Reviewing What Is Your Position 
problem/Discussion of calculation of slope from 
two points 

□ 

□ 

□ 

□ 

d. 

to give each step as much attention as you would like. 
Finish and review Which Form is Best? 

not do 
□ 

only a 

□ 

□ 

e. 

Reviewing Equivalent Forms sheet (7.2) 

□ 

□ 

□ 

□ 


25. Did you use Lesson 8? 

Yes 

No 


□ 

□ 

If your answer is No, skip questions 26 and 27, then go to question 28. 

26. Lesson 8 was anticipated to take 150 minutes (3 


periods). Approximately how many minutes did 
you spend on Lesson 8? 


minutes 


27. 

Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Please tell us how much attention you gave to each 
step when teaching Lesson 8. 

Did 

not do 

Did and gave this step . . . 

only a 

little some a lot of 

attention attention attention 

a. 

Showing computer simulation 

□ 

□ 

□ 

□ 

b. 

Introducing the 5 plane problem 

□ 

□ 

□ 

□ 

c. 

Students work in groups to solve 

□ 

□ 

□ 

□ 

d. 

Group presentations on possible solutions 
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Algebra I Implementation Survey: Air Traffic Control (RCT) 

28. Overall, across all lessons, to what extent was each of the 
following things a challenge for you when teaching the Air 
Traffic Control unit? 


Not a 
challenge 


Appendix C 


A major 
challenge 




*- 


a. 

The amount of content in the unit 

□ 

□ 

□ 

□ 

□ 

b. 

The difficulty of the ideas and concepts in the unit 

□ 

□ 

□ 

□ 

□ 

c. 

The amount of time it took to complete the unit 

□ 

□ 

□ 

□ 

□ 

d. 

The classroom management demands of the unit 

□ 

□ 

□ 

□ 

□ 

e. 

The physical materials management demands of the unit 

□ 

□ 

□ 

□ 

□ 

f. 

Motivating students to engage seriously in the unit 

□ 

□ 

□ 

□ 

□ 

& 

Students with adequate reading and writing skills for the unit 

□ 

□ 

□ 

□ 

□ 

h. 

Students with adequate prior math skills for the unit 

□ 

□ 

□ 

□ 

□ 

i. 

Students with adequate self-management skills for the unit 

□ 

□ 

□ 

□ 

□ 

j- 

Students with adequate teamwork skills for the unit 

□ 

□ 

□ 

□ 

□ 

k. 

Students with adequate problem-solving skills for the unit 

□ 

□ 

□ 

□ 

□ 

1 . 

Too many students in the class (es) 

□ 

□ 

□ 

□ 

□ 


29. Please provide any other information about teaching this unit that you think would be useful. Use an 
additional page if necessary. 
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Algebra I Implementation Survey: Air Traffic Control (RCT) 

28. Overall, across all lessons, to what extent was each of the 


Not a 


Appendix C 


A major 


Questions 30-32 pertain to the weekly online Coaching & Collaboration support provided for this 

program. 


30. How many hours of Coaching & Collaboration did you participate in for this unit? 


Coaching & Collaboration Hours 
for Air Traffic Control 

Hours 
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Algebra I Implementation Survey: Air Traffic Control (RCT) 


31. 

Please tell us how you spent your Coaching & 

Did 

Did and gave this step . 



Collaboration time for Air Traffic Control. 

not do 

only a 

some 

a lot of 




little time 

time 

time 

a. 

Discussing implementation of the lessons 

□ 

□ 

□ 

□ 

b. 

Discussing common student misconceptions 

□ 

□ 

□ 

□ 

c. 

Reviewing student work and determining next steps 

□ 

□ 

□ 

□ 

d. 

Discussing math concepts used in the unit 

□ 

□ 

□ 

□ 

e. 

Discussing ways of engaging students 

□ 

□ 

□ 

□ 

f. 

Discussing ways to connect Drop-in Unit to textbook 

□ 

□ 

□ 

□ 

g- 

Discussing ways to enrich textbook lessons 

□ 

□ 

□ 

□ 

h. 

Discussing ways to assess student learning 

□ 

□ 

□ 

□ 


32. Please provide any other information about the Coaching & Collaboration support for this unit that you think 
would be useful. Use an additional page if necessary. 
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Algebra I Implementation Survey: Air Traffic Control (RCT) 

Questions 33-38 pertain to the College Awareness Curriculum (CAC) for this unit. 

33. How much time (number of minutes] did students spend engaged in College Awareness Curriculum (CAC) 
activities during this unit? 


Total Number of Minutes Spent on 
CAC Activities This Unit 

minutes 


Please report on the two CAC activities/modules that you used. (Note: You may have implemented more than 
2 CAC activities for this unit, however, we only require you to report on two.) 


34. Name of one CAC activity/module used to 
reinforce instruction during this unit: 
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Algebra I Implementation Survey: Air Traffic Control (RCT) 

35 To what extent was each of the following a challenge for 
. you when teaching the CAC activity you named in 

question 34? Not a A major 

challenge challenge 


-* 


*• 


a. 

Relevance of the content of the activity to middle school 
students 

□ 

□ 

□ 

□ 

□ 

b. 

Completing the activity within the 15 minute limit 

□ 

□ 

□ 

□ 

□ 

c. 

Motivating students to engage in the activity 

□ 

□ 

□ 

□ 

□ 

d. 

Integrating the activity with math instruction 

□ 

□ 

□ 

□ 

□ 

e. 

Too many students in the class(es] 

□ 

□ 

□ 

□ 

□ 


36. Name of another CAC activity/module used 
to reinforce instruction during this unit: 


37. To what extent was each of the following a challenge for you 
when teaching the CAC activity you named in question 36? 


Not a 
challenge 


A major 
challenge 
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◄- 




a. 

Relevance of the content of the activity to middle school 
students 


□ 


□ 


□ 


□ 


□ 

b. 

Completing the activity within the 15 minute limit 

□ 


□ 


□ 


□ 


□ 


c. 

Motivating students to engage in the activity 


□ 


□ 


□ 


□ 


□ 

d. 

Integrating the activity with math instruction 

□ 


□ 


□ 


□ 


□ 


e. 

Too many students in the class (es) 


□ 


□ 


□ 


□ 


□ 


38. Please provide any other information about the College Awareness Curriculum (CAC] activities 
that you think would be useful. Use an additional page if necessary. 


THANK YOU VERY MUCH FOR COMPLETING THIS SURVEY! 


Please place your completed survey in the business reply envelope 
(provided) and place the envelope in the mail (no postage is 
necessary). We appreciate you returning this survey as soon as you 

complete it. 
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This survey is being administered by WestEd as part of the STEM Learning Opportunities Providing Equity 
project. This project is funded by the U. S. Department of Education. 
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Dear teacher, 

Thank you in advance for completing this survey! The purpose of this survey is to gather information on the 
implementation of the project-based math curriculum, the related Coaching & Collaboration, and the College 
Awareness Curriculum [CAC], All data you provide will remain confidential and will be used solely for this 
research study. No data you provide will be shared with your school principal or other school/district 
administrators or staff. If you have any questions about this survey, please contact Carol Whang at (415] 615- 
3346 or cwhang@wested.org. 


1. Name: Please provide your first name, middle initial, and last name below. 


First 

M.I. 

Last 





2. Contact Information: Please provide your email address and phone number. We will only use this 
information to contact you if we have questions about any information you submit for this study. 


Email address 

Phone Number 


( ) 


3. Date you started teaching the Air Traffic Control unit (e.g. 9/3/07]: / / 

The Air Traffic Control unit has 8 lessons. Questions 4-29 pertain to the Air Traffic Control lessons. 

4. Did you use Lesson 1? Yes No 

□ □ 

If your answer is No, skip questions 5 and 6, then go to question 7. 

5. Lesson 1 was anticipated to take 100 minutes 
(2 periods]. Approximately how many minutes 

... . minutes 

did you spend on Lesson 1? 
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6. Often, it is impossible to do all the steps in a lesson or Did Did and gave this step . . . 



to give each step as much attention as you would like. 

Please tell us how much attention you gave to each 
step when teaching Lesson 1. 

not do 

only a 
little 

attention 

some 

attention 

a lot of 

attention 

a. 

Introducing the unit and showing videos 

□ 

□ 

□ 

□ 

b. 

Working through Delta 134 problem 

□ 

□ 

□ 

□ 

c. 

Working through Southwest 190 problem 

□ 

□ 

□ 

□ 

d. 

Working through American 722 problem 

□ 

□ 

□ 

□ 

e. 

Discussion of x- and y-intercept 

□ 

□ 

□ 

□ 

7. 

Did you use Lesson 2? Yes 


No 




□ 


□ 




If your answer is No, skip questions 8 and 9, then go to question 10. 




8. 

Lesson 2 was anticipated to take 75 minutes (1.5 






periods]. Approximately how many minutes did 



minutes 



you spend on Lesson 2? 





9. 

Often, it is impossible to do all the steps in a lesson or 

Did 

Did and gave this step . . . 


to give each step as much attention as you would like. 

not do 

only a 




Please tell us how much attention you gave to each 


little 

some 

a lot of 


step when teaching Lesson 2. 


attention 

attention 

attention 

a. 

Class discussion on rate of change 

□ 

□ 

□ 

□ 

b. 

Student pairs completing Southwest 190 problem 

□ 

□ 

□ 

□ 

c. 

Reviewing graphing rate of change homework 

□ 

□ 

□ 

□ 


Did you use Lesson 3? 

Yes 

No 


□ 

□ 

If your answer is No, skip questions 11 and 12, then go to question 13. 


Lesson 3 was anticipated to take 100 minutes (2 

periods]. Approximately how many minutes did 
you spend on Lesson 3? 


minutes 


12. Often, it is impossible to do all the steps in a lesson or Did 
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to give each step as much attention as you would like. 

not do 

only 




Please tell us how much attention you gave to each 


a 

some 

a lot of 


step when teaching Lesson 3. 


attention 

attention 

attention 

a. 

Direct instruction on slope 





b. 

Opening class work Delta 134 problem 





c. 

Student pairs completing Delta 134 problem 





d. 

Reviewing calculating slopes homework 





e. 

Creation and display of slope/rate of change posters 





13. 

Did you use Lesson 4? 

Yes 


No 




□ 


□ 



If your answer is No, skip questions 14 and 15, then go to question 16. 




14. 

Lesson 4 was anticipated to take 75 minutes (1.5 
periods]. Approximately how many minutes did 
you spend on Lesson 4? 



minutes 







15. 

Often, it is impossible to do all the steps in a lesson or 

Did 

Did and gave this step 



to give each step as much attention as you would like. 

not do 

only a 




Please tell us how much attention you gave to each 


little 

some 

a lot of 


step when teaching Lesson 4. 


attention 

attention 

attention 

a. 

Discussion of constant distance 

□ 

□ 

□ 

□ 

b. 

Student pairs completing American 722 problem 

□ 

□ 

□ 

□ 

c. 

Reviewing parallel lines homework 

□ 

□ 

□ 

□ 

16. 

Did you use Lesson 5? 

Yes 


No 




□ 


□ 



If your answer is No, skip questions 17 and 18, then go to question 19. 




17. 

Lesson 5 was anticipated to take 75 minutes (1.5 
periods]. Approximately how many minutes did 



minutes 



you spend on Lesson 5? 


UUestEd 


61 


Appendix D 

STEM Learning Opportunities Providing Equity 

Algebra I Implementation Survey: Air Traffic Control (RCT) 


18. 

Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 
Please tell us how much attention you gave to each 
step when teaching Lesson 5. 

Did 

not do 

Did and gave this step 

only a 

little some 

attention attention 

a lot of 

attention 

a. 

Introduction to Safe Landings problem 





b. 

Individual work on Safe Landings problem 





c. 

Reviewing different solutions to Safe Landings 





19. 

Did you use Lesson 6? 

Yes 

□ 


No 

□ 



If your answer is No, skip questions 20 and 21, then go to question 22. 




20. 

Lesson 6 was anticipated to take 150 minutes (3 
periods]. Approximately how many minutes did 



minutes 



you spend on Lesson 6? 





21. 

Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Please tell us how much attention you gave to each 
step when teaching Lesson 6. 

Did 

not do 

Did and gave this step 

only a 

little some 

attention attention 

a lot of 

attention 

a. 

Review of previous in Lesson Introduction 

□ 

□ 

□ 

□ 

b. 

Discussion on slope and linear equation connection 

□ 

□ 

□ 

□ 

c. 

Reviewing What Is Your Position 
problem/Discussion of calculation of slope from 
two points 

□ 

□ 

□ 

□ 

d. 

Introduction to Big Picture /two variable problem 

□ 

□ 

□ 

□ 

e. 

Reviewing Big Picture problem 

□ 

□ 

□ 

□ 

f. 

Student work on graphing equations sheet (6.3) 

□ 

□ 

□ 

□ 

g- 

Reviewing graphing equations sheet (6.3) 

□ 

□ 

□ 

□ 

22. 

Did you use Lesson 7? 

Yes 

□ 


No 

□ 



If your answer is No, skip questions 23 and 24, then go to question 25. 




23. 

Lesson 7 was anticipated to take 75 minutes (1.5 
periods). Approximately how many minutes did 



minutes 



you spend on Lesson 7? 
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24. 

Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 
Please tell us how much attention you gave to each 
step when teaching Lesson 7. 

Did 

not do 

Did and gave this step 

only a 

little some 

attention attention 

a lot of 

attention 

a. 

Independent work on Which Form Is Best? 
introduction 

□ 

□ 


□ 

□ 

b. 

Discussion on slope-intercept vs. standard form 

□ 

□ 


□ 

□ 

c. 

Reviewing What Is Your Position 
problem/Discussion of calculation of slope from 
two points 

□ 

□ 


□ 

□ 

d. 

Finish and review Which Form is Best? 

□ 

□ 


□ 

□ 

e. 

Reviewing Equivalent Forms sheet (7.2) 

□ 

□ 


□ 

□ 

25. 

Did you use Lesson 8? 

Yes 

□ 


No 

□ 




If your answer is No, skip questions 26 and 27, then go to question 28. 





26. 

Lesson 8 was anticipated to take 150 minutes (3 
periods). Approximately how many minutes did 



minutes 



you spend on Lesson 8? 






27. 

Often, it is impossible to do all the steps in a lesson or 
to give each step as much attention as you would like. 

Please tell us how much attention you gave to each 
step when teaching Lesson 8. 

Did 

not do 

Did and gave this step 

only a 

little some 

attention attention 

a lot of 

attention 

a. 

Showing computer simulation 

□ 

□ 


□ 

□ 

b. 

Introducing the 5 plane problem 

□ 

□ 


□ 

□ 

c. 

Students work in groups to solve 

□ 

□ 


□ 

□ 


d. Group presentations on possible solutions 
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STEM Learning Opportunities Providing Equity 

Algebra I Implementation Survey: Air Traffic Control (RCT) 


28. Overall, across all lessons, to what extent was each of the Not a Ama'or 

following things a challenge for you when teaching the Air , 3 , major 

challenge challenge 

Traffic Control unit? 


◄ ► 


a. 

The amount of content in the unit 

□ 

□ 

□ 

□ 

□ 

b. 

The difficulty of the ideas and concepts in the unit 

□ 

□ 

□ 

□ 

□ 

c. 

The amount of time it took to complete the unit 

□ 

□ 

□ 

□ 

□ 

d. 

The classroom management demands of the unit 

□ 

□ 

□ 

□ 

□ 

e. 

The physical materials management demands of the unit 

□ 

□ 

□ 

□ 

□ 

f. 

Motivating students to engage seriously in the unit 

□ 

□ 

□ 

□ 

□ 

g- 

Students with adequate reading and writing skills for the unit 

□ 

□ 

□ 

□ 

□ 

h. 

Students with adequate prior math skills for the unit 

□ 

□ 

□ 

□ 

□ 

i. 

Students with adequate self-management skills for the unit 

□ 

□ 

□ 

□ 

□ 

)■ 

Students with adequate teamwork skills for the unit 

□ 

□ 

□ 

□ 

□ 

k. 

Students with adequate problem-solving skills for the unit 

□ 

□ 

□ 

□ 

□ 

1 . 

Too many students in the class(es] 

□ 

□ 

□ 

□ 

□ 


29. Please provide any other information about teaching this unit that you think would be useful. Use an 
additional page if necessary. 


Questions 30-32 pertain to the weekly online Coaching & Collaboration support provided for this 

program. 

30. How many hours of Coaching & Collaboration did you participate in for this unit? 

Coaching & Collaboration Hours 
for Air Traffic Control 

Hours 
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31. 

Please tell us how you spent your Coaching & 
Collaboration time for Air Traffic Control. 

Did 

not do 

Did and gave this step . 

only a some 

little time time 

a lot of 

time 

a. 

Discussing implementation of the lessons 

□ 

□ 

□ 

□ 

b. 

Discussing common student misconceptions 

□ 

□ 

□ 

□ 

c. 

Reviewing student work and determining next steps 

□ 

□ 

□ 

□ 

d. 

Discussing math concepts used in the unit 

□ 

□ 

□ 

□ 

e. 

Discussing ways of engaging students 

□ 

□ 

□ 

□ 

f. 

Discussing ways to connect Drop-in Unit to textbook 

□ 

□ 

□ 

□ 

g- 

Discussing ways to enrich textbook lessons 

□ 

□ 

□ 

□ 

h. 

Discussing ways to assess student learning 

□ 

□ 

□ 

□ 


32. Please provide any other information about the Coaching & Collaboration support for this unit that you think 
would be useful. Use an additional page if necessary. 


Questions 33-38 pertain to the College Awareness Curriculum (CAC) for this unit. 

33. How much time (number of minutes] did students spend engaged in College Awareness Curriculum (CAC] 
activities during this unit? 

Total Number of Minutes Spent on 
CAC Activities This Unit 

minutes 


Please report on the two CAC activities/modules that you used. (Note: You may have implemented more than 
2 CAC activities for this unit, however, we only require you to report on two.] 

34. Name of one CAC activity/module used to 

reinforce instruction during this unit: 
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35 To what extent was each of the following a challenge for 
. you when teaching the CAC activity you named in Not a A major 

question 34? challenge challenge 

◄ ► 


a. 

Relevance of the content of the activity to middle school 
students 

□ 

□ 

□ 

□ 

□ 

b. 

Completing the activity within the 15 minute limit 

□ 

□ 

□ 

□ 

□ 

c. 

Motivating students to engage in the activity 

□ 

□ 

□ 

□ 

□ 

d. 

Integrating the activity with math instruction 

□ 

□ 

□ 

□ 

□ 

e. 

Too many students in the class [es] 

□ 

□ 

□ 

□ 

□ 


36. Name of another CAC activity/module used 

to reinforce instruction during this unit: 

37. To what extent was each of the following a challenge for you Not a A major 

when teaching the CAC activity you named in question 36? challenge challenge 

◄ ► 


a. 

Relevance of the content of the activity to middle school 
students 

□ 

□ 

□ 

□ 

□ 

b. 

Completing the activity within the 15 minute limit 

□ 

□ 

□ 

□ 

□ 

c. 

Motivating students to engage in the activity 

□ 

□ 

□ 

□ 

□ 

d. 

Integrating the activity with math instruction 

□ 

□ 

□ 

□ 

□ 

e. 

Too many students in the class(es] 

□ 

□ 

□ 

□ 

□ 


38. Please provide any other information about the College Awareness Curriculum (CAC] activities that you think 
would be useful. Use an additional page if necessary. 
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THANK YOU VERY MUCH FOR COMPLETING THIS SURVEY! 

Please place your completed survey in the business reply envelope 
(provided) and place the envelope in the mail (no postage is necessary). We 
appreciate you returning this survey as soon as you complete it. 
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2012-2013 Measuring Fidelity of Implementation for Algebra I Drop-in Units: DEV11 (SLOPE) 


Component 

Description 

Operational 

Definition 

Data Collection 
Measure 

Possible 

Implementation 

Range 

Expected 
Teacher-level 
Implementation 
with Fidelity / 
Corresponding 
Score 

Teacher-level 
Composite Score / 
Criterion for 
Adequate 
(implementation 
"with fidelity") 

Program-level Criterion for 
Adequate 

(implementation "with 
fidelity") 

Considered 

"with 

fidelity" for 
Year 1 

Component I. Curricular Materials 

1 

Year 1 Materials 

Teacher 
receives all 
curriculum 
materials for 
intervention 

Receipt 

verification from 

implementation 

team 

No or Yes 

Received materials 
/ No or Yes: 0-1 

0-1 

Adequate: 1 

1 00% of teachers meet 
criterion 

No or Yes 

Component II. Algebra I PD-Year 1 (online) 

2 

Year 1 PD for 
Algebra 1 (3 
half-day online 
sessions) for all 
treatment 

teachers 

Teacher 
attends training 
for 

implementing 

C onnectEd 
Algebra I 
curriculum 

Attendance 
record/log from 
implementation 
team 

0-12 hours 

Attended all 12 
hours / No or Yes: 
0-1 

0-2 

Adequate: 2 

80% of teachers met criterion 

No or Yes 

3 

Year 1 

Orientation 
workshop for 
Algebra I 

College 

Awareness (4 
hours) for all 
treatment 

teachers 

Teacher 
attends training 
for 

implementing 
RTF college 
awareness 

curriculum 

Attendance 
record/log from 
implementation 
team 

0-4 hours 

Participated / No 
or Yes: 0-1 
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Component III. Coaching (online) 


4 

Y1 Algebra 1 
Coaching and 
Collaboration for 
all treatment 
teachers (online 
sessions 

scheduled for 1-3 
hours/week): 

Unit 1 

Teacher 
discusses at 
least one issue 
with coach 
and/or interacts 
with coach and 
other teachers 

Algebra I 
Implementation 
Survey for 

Puzzle Cube 
(Questions 19- 
21), and/or 
coaching log 

0-3 hrs per week 

Participated at 
least 1 hour per 
week / No or Yes: 
0-1 

0-3 

Adequate: 3 

80% of teachers met criterion 

No or Yes 

5 

Y1 Algebra 1 
Coaching and 
Collaboration for 
all treatment 
teachers (online 
sessions 

scheduled for 1-3 
hours/week): 

Unit 2 

Teacher 
discusses at 
least one issue 
with coach 
and/or interacts 
with coach and 
other teachers 

Algebra I 
Implementation 
Survey for Air 
Traffic Control 
(Questions 28- 
30), and/or 
coaching log 

0-3 hrs per week 

Participated at 
least 1 hour per 
week / No or Yes: 
0-1 




6 

Y1 Algebra 1 
Coaching and 
Collaboration for 
all treatment 
teachers (online 
sessions 

scheduled for 1-3 
hours/week): 

Unit 3 

Teacher 
discusses at 
least one issue 
with coach 
and/or interacts 
with coach and 
other teachers 

Algebra I 
Implementation 
Survey for 

Catapult Game 
(Questions 19- 
21), and/or 
coaching log 

0-3 hrs per week 

Participated at 
least 1 hour per 
week / No or Yes: 
0-1 
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Components IV-a, IV-b, IV-c. Teacher Implementation of Classroom Intervention (Algebra I Units 1, 2, and 3 Respectively) 



Component: 

3 Algebra I 
units 

Operational 

Definition 

Data Collection 
Measure 

Possible 

Implementation 

Range 

Expected 
Teacher-level 
Implementation 
with Fidelity / 
Corresponding 
Score 

Teacher-level 
Composite Score / 
Criterion for 
Adequate 
(implementation 
"with fidelity") 

Program-level Criterion for 
Adequate 

(implementation "with 
fidelity") 

Considered 

"with 

fidelity" for 
Year 1 

7 

IV- 

a 

Y1 Algebra I: 
Implementation 
of Algebra Drop- 
in Unit 1 

Teacher 

implements 

ConnectEd 

math 

curriculum as 
trained in PD 

Algebra I 
Implementation 
Survey for 

Puzzle Cube 
(Questions 2-18) 






a. Number of 
lessons used 


a. 0-5 lessons 

a. > 80% of 
implementation 
range (i.e., 4 
lessons) 

/ No or Yes: 0-1 

0-16 

Adequate: > 75% 
of possible highest 
composite score; 

i.e., the score for 
implementation 
with fidelity is at 
least 12 

(this indicates that 
the minimal 
number of lessons 
to be taught is 4; 
no way to reach a 
score of 12 or 
above if teachers 
cover 3 or less 
lessons) 

80% of teachers meet criterion 

No or Yes 

b. Time spent 
on each lesson 


b. 0-150 minutes 
(varies for each 
lesson) 

b. > 70% of 
implementation 
range for each 
taught lesson 
/ No or Yes: 0-1 
for each taught 
lesson 

(0-5 across all 
lessons) 

c. Number of 
steps used 
within each 
lesson 


c. 0-8 steps 
(varies for each 
lesson) 

b. > 70% of 
implementation 
range for each 
taught lesson / No 
or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons) 
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d. Average 
level of 
attention given 
to used steps 
within each 
lesson 


d. 1-3 (1 = gave 
only a little 
attention; 3 = gave 
a lot of attention) 

d. > 2 for each 
taught lesson / No 
or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons) 




8 

IV- 

b 

Y1 Algebra I: 
Implementation 
of Algebra Drop- 
in Unit 2 

Teacher 

implements 

ConnectEd 

math 

curriculum as 
trained in PD 

Algebra I 
Implementation 
Survey for Air 
Traffic Control 
(Questions 2-27) 






a. Number of 
lessons used 


a. 0-8 lessons 

a. > 75% of 
implementation 
range (i.e., 6 
lessons) 

/ No or Yes: 0-1 

0-25 

Adequate: > 72% 
of possible highest 
composite score; 
i.e., the score for 
implementation 
with fidelity is at 
least 18 

(this indicates that 
the minimal 
number of lessons 
to be taught is 6; 
no way to reach a 
score of 18 or 
above if teachers 
cover 5 or less 
lessons) 

80% of teachers meet criterion 

No or Yes 

b. Time spent 
on each lesson 


b. 0-150 minutes 
(varies for each 
lesson) 

b. > 70% of 
implementation 
range for each 
taught lesson 
/ No or Yes: 0-1 
for each taught 
lesson 

(0-8 across all 
lessons) 

c. Number of 
steps used 
within each 
lesson 


c. 0-7 steps 
(varies for each 
lesson) 

b. > 70% of 
implementation 
range for each 
taught lesson 
/ No or Yes: 0-1 
for each taught 
lesson 

(0-8 across all 
lessons) 

d. Average 
level of 
attention given 
to each lesson 


d. 1-3 (1 = gave 
only a little 
attention; 3 = gave 
a lot of attention) 

d. > 2 for each 
taught lesson / No 
or Yes: 0-1 for 
each taught lesson 
(0-8 across all 
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lessons) 




9 

IV- 

c 

Y1 Algebra I: 
Implementation 
of Algebra Drop- 
in Unit 3 

Teacher 

implements 

ConnectEd 

math 

curriculum as 
trained in PD 

Algebra I 
Implementation 
Survey for 

Catapult Game 
(Questions 2-18) 








a. Number of 
lessons used 


a. 0-5 lessons 

a. > 80% of 
implementation 
range (i.e., 4 
lessons) 

/ No or Yes: 0-1 

0-16 

Adequate: > 75% 
of possible highest 
composite score; 
i.e., the score for 

80% of teachers meet criterion 

No or Yes 



b. Time spent 
on each lesson 
/ time spent 
across all 
lessons 


b. 0-200 minutes 
(varies for each 
lesson) 

b. > 70% of 
implementation 
range for each 
taught lesson 
/ No or Yes: 0-1 
for each taught 
lesson 

(0-5 across all 
lessons) 

implementation 
with fidelity is at 
least 12 

(this indicates that 
the minimal 
number of lessons 
to be taught is 4; 
no way to reach a 
score of 12 or 





c. Number of 
steps used 
within each 
lesson / 
number of 
steps used 
across all 
lessons 


c. 0-9 steps 
(varies for each 
lesson) 

b. > 70% of 
implementation 
range for each 
taught lesson 
/ No or Yes: 0-1 
for each taught 
lesson 

(0-5 across all 
lessons) 

above if teachers 
cover 3 or less 
lessons) 





d. Average 
level of 
attention given 
to each lesson / 
average level 
of attention 


d. 1-3 (1 = gave 
only a little 
attention; 3 = gave 
a lot of attention) 

d. > 2 for each 
taught lesson / No 
or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons) 
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Components IV-d, IV-e, IV-f. College Awareness Curriculum (Units 1, 2, and 3 Respectively) 


Components 

College 

Awareness 

Curriculum 

units 

Operational 

Definition 

Data Collection 
Measure 

Possible 

Implementation 

Range 

Expected 
Teacher-level 
Implementation 
with Fidelity / 
Corresponding 
Score 

Teacher-level 
Composite Score / 
Criterion for 
Adequate 
(implementation 
"with fidelity") 

Program-level Criterion for 
Adequate(implementation 
"with fidelity") 

Considered 

"with 

fidelity" for 
Year 1 

10 

IV- 

d 

Y1 Algebra I: 
Implementation 
ofCAC Unit 1 

Teacher 

implements 

RTF college 
awareness 
curriculum (2 
activities) as 
trained in PD 

Algebra I 
Implementation 
Survey for 

Puzzle Cube 
(Questions 22- 
27) 

0-30 minutes 
(no more than 1 5 
minutes per 
activity) 

<30 minutes / 

No or Yes: 0-1 

0-3 

Adequate: 3 

80% of teachers meet criterion 

No or Yes 

11 

IV- 

e 

Y1 Algebra I: 
Implementation 
ofCAC Unit 2 

Teacher 

implements 

RTF college 
awareness 
curriculum (2 
activities) as 
trained in PD 

Algebra I 
Implementation 
Survey for Air 
Traffic Control 
(Questions 31- 
36) 

0-30 minutes 
(no more than 1 5 
minutes per 
activity) 

<30 minutes / 

No or Yes: 0-1 

12 

IV- 

f 

Y1 Algebra I: 
Implementation 
ofCAC Unit 3 

Teacher 

implements 

RTF college 
awareness 
curriculum (2 
activities) as 
trained in PD 

Algebra 1 
Implementation 
Survey for 

Catapult Game 
(Questions 22- 
27) 

0-30 minutes 
(no more than 1 5 
minutes per 
activity) 

<30 minutes / 

No or Yes: 0-1 
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Findings from Evaluator Study of Implementation: IMPLEMENTATION YEAR 1 

Enter calendar year: Summer 2012 plus September 2012-June 2013 

Intervention 
Components: 
Copy from list 
above 

Implementation 
measure (total number 
of measurable 
indicators 
representing each 
component 

Sample 
Size at the 
Sample 
Level (# of 
schools, 
districts, 
etc) 

Representativeness 
of sample: Measured 
on All (A), Some (S), 
or None (N) of the 
units representing 
the intervention 
group in the impact 
analyses 6 

Component Level 
Threshold for Fidelity of 
Implementation for the 
Unit that is the Basis for 
the Sample-Level 

Evaluator’s 
Criteria for 
“Implemented 
with Fidelity” 
at Sample 
Level 

Component 
Level Fidelity 
Score for the 
Entire 
Sample 

Implemented 
with Fidelity? 

(Yes, No, N/A) 

Planned Intervention Activities [i.e., key components] 

C.A.M.P. 

Curricular 

Materials 

Teacher receives all 
curriculum materials 

for intervention 

24 

treatment 

teachers 

A 

Received materials / No 
or Yes: 0-1 

100% 

100% 

Yes 

C.A.M.P. 

Professional 

Development 

Year 1 PD for 

Summer C.A.M.P. (5 
full-day meetings) for 
treatment teachers 

Year 1 Orientation 
workshop for 

Summer C.A.M.P. 
College Awareness 
(4 hours) for all 
treatment College 
Awareness 

Instructors 

24 

treatment 

teachers 

A 

Teachers participated in 
all 5 days of PD for 
Summer C.A.M.P. 

AND 

College Awareness 
instructors participated 
in all four hours of CAC 
training. 

80% 

100% 

Yes 

C.A.M.P. 

Coaching 

Y1 Pre-Algebra 
Coaching and 
Collaboration for all 

treatment teachers 
(online sessions 
scheduled for 1 
hour/week): Units 1-3 

23 

treatment 

teachers 

S 

Teachers participated in 
a minimum of one hour 
of Coaching/ 
Collaboration time per 
week for EACH unit 
(units 1-3) 

80% 

78% 

No 

C.A.M.P. 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 1 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 
lessons)) 

24 

treatment 

teachers 

A 

Adequate: >= 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 
teachers cover 3 or less 
lessons) 

80% 

79% 

No 


76 



Appendix G 



c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-5 across all 
lessons)) 







C.A.M.P. 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 2 

a. Number of lessons 
used (> 70% of 
implementation range 
i.e., 5 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-7 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-7 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 

23 

treatment 

teachers 

S 

Adequate: >= 72% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 16 
(this indicates that the 
minimal number of 
lessons to be taught is 

5; no way to reach a 
score of 16 or above if 
teachers cover 4 or less 
lessons) 

80% 

61% 

No 
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taught lesson / No or 
Yes: 0-1 for each 
taught lesson 
(0-7 across all 
lessons)) 







C.A.M.P. 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 3 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-5 across all 
lessons)) 

23 

treatment 

teachers 

S 

Adequate: >= 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 
teachers cover 3 or less 
lessons) 

80% 

39% 

No 

C.A.M.P. 

College 

Awareness 

Curriculum 

Teacher implements 
RTF college 

awareness 

curriculum as trained 

in PD: Units 1-3 

24 

treatment 

teachers 

A 

All three CAC units are 
presented in treatment 
classrooms 

80% 

74% 

No 

Algebra 1 

Curricular 

Materials 

Teacher receives all 

curriculum materials 
for intervention 

28 

treatment 

teachers 

A 

Received materials / No 
or Yes: 0-1 

100% 

100% 

Yes 
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Algebra 1 

Professional 

Development 

Teacher attends 
training for 
implementing 
ConnectEd Algebra 1 
curriculum 

Teacher attends 
training for 
implementing RTF 
college awareness 
curriculum 

28 

treatment 

teachers 

A 

Teachers participated in 
all 5 days of PD for 
Summer C.A.M.P. 

AND 

participated in all four 
hours of CAC training. 

80% 

96% 

Yes 

Algebra 1 
Coaching 

Y1 Algebra 1 

Coaching and 
Collaboration for all 
treatment teachers 
(online sessions 
scheduled for 1-3 
hours/week): 

Units 1-3 

28 

treatment 

teachers 

A 

Teachers participated in 
a minimum of one hour 
of Coaching/ 
Collaboration time per 
week for EACPI unit 
(units 1-3) 

80% 

57% 

No 

Algebra 1 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 1 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 

28 

treatment 

teachers 

A 

Adequate: > 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 
teachers cover 3 or less 
lessons) 

80% 

89% 

Yes 
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taught lesson / No or 
Yes: 0-1 for each 
taught lesson 
(0-5 across all 
lessons)) 







Algebra 1 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 2 

a. Number of lessons 
used (> 75% of 
implementation range 
i.e., 6 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-8 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-8 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-8 across all 
lessons)) 

28 

treatment 

teachers 

A 

Adequate: > 72% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 18 
(this indicates that the 
minimal number of 
lessons to be taught is 

6; no way to reach a 
score of 1 8 or above if 
teachers cover 5 or less 
lessons) 

80% 

64% 

No 

Algebra 1 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 3 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 

28 

treatment 

teachers 

A 

Adequate: > 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 

80% 

75% 

No 
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lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-5 across all 
lessons)) 



teachers cover 3 or less 
lessons) 




Algebra 1 

College 

Awareness 

Curriculum 

Teacher implements 
RTF college 
awareness 

curriculum (2 
activities) as trained 
in PD: Units 1-3 

28 

treatment 

teachers 

A 

Two activities (fora 
maximum of 30 
minutes) are 
implemented in the 
treatment classrooms. 

80% 

29% 

No 

a Mediators refer to measurement of short-term outcomes, e.g., changes in student/teacher/parent behaviors or attitudes that are 
assumed to have a direct connection to long-term outcomes. If these mediators have been measured in both the treatment and 
comparison groups and are being reported as part of an exploratory contrast, they should not be entered here. If these mediators 
have been measured in the treatment group only or in both the treatment and the comparison group and are not being reported as 
part of an exploratory contrast, the findings for just the treatment group would be reported here. If evaluators believe they have 
findings on mediators to report, they should consult with their TA Liaison or the AR helpdesk to confirm that the findings are 
appropriate for this section of the Survey. 

b All: If the intervention group in the impact analysis includes 10 schools and fidelity measurement includes these 10 schools, the 
evaluator would enter “A” indicating that All of the schools in the impact analysis are represented in the fidelity findings. Some: If the 
intervention group in the impact analysis includes teachers in grades K to 3 but fidelity is measured only for teachers in Kindergarten, 
the evaluator would enter “S” indicating that Some of the teachers in the impact analysis are represented in the fidelity findings. None: 

If the intervention group in the impact analysis includes grades 7 - 9 but fidelity is measured only for grades 5-6, the evaluator would 
enter “N” indicating that None of the grades in the impact analysis are represented in the fidelity findings. 
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Findings from Evaluator Study of Implementation: IMPLEMENTATION YEAR 2 

Enter calendar year: Summer 2013 plus September 2013-June 2014 

Intervention 
Components: 
Copy from list 
above 

Implementation 
measure (total number 
of measurable 
indicators 
representing each 
component 

Sample 
Size at the 
Sample 
Level (# of 
schools, 
districts, 
etc) 

Representativeness 
of sample: Measured 
on All (A), Some (S), 
or None (N) of the 
units representing 
the intervention 
group in the impact 
analyses 6 

Component Level 
Threshold for Fidelity of 
Implementation for the 
Unit that is the Basis for 
the Sample-Level 

Evaluator’s 
Criteria for 
“Implemented 
with Fidelity” 
at Sample 
Level 

Component 
Level Fidelity 
Score for the 
Entire 
Sample 

Implemented 
with Fidelity? 

(Yes, No, N/A) 

Planned Intervention Activities [i.e., key components] 

C.A.M.P. 

Curricular 

Materials 

Teacher receives all 
curriculum materials 

for intervention 

14 

treatment 

teachers 

S 

Received materials / No 
or Yes: 0-1 

100% 

100% 

Yes 

C.A.M.P. 

Professional 

Development 

Year 1 PD for 

Summer C.A.M.P. (5 
full-day meetings) for 
treatment teachers 

Year 1 Orientation 
workshop for 

Summer C.A.M.P. 
College Awareness 
(4 hours) for all 
treatment College 
Awareness 

Instructors 

14 

treatment 

teachers 

S 

Teachers participated in 
all 5 days of PD for 
Summer C.A.M.P. 

AND 

College Awareness 
instructors participated 
in all four hours of CAC 
training. 

80% 

100% 

Yes 

C.A.M.P. 

Coaching 

Y1 Pre-Algebra 
Coaching and 
Collaboration for all 

treatment teachers 
(online sessions 
scheduled for 1 
hour/week): Units 1-3 

14 

treatment 

teachers 

S 

Teachers participated in 
a minimum of one hour 
of Coaching/ 
Collaboration time per 
week for EACH unit 
(units 1-3) 

80% 

36% 

No 

C.A.M.P. 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 1 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 

14 

treatment 

teachers 

S 

Adequate: >= 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 
teachers cover 3 or less 
lessons) 

80% 

21% 

No 
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lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-5 across all 
lessons)) 







C.A.M.P. 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 2 

a. Number of lessons 
used (> 70% of 
implementation range 
i.e., 5 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-7 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-7 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 

14 

treatment 

teachers 

S 

Adequate: >= 72% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 16 
(this indicates that the 
minimal number of 
lessons to be taught is 

5; no way to reach a 
score of 16 or above if 
teachers cover 4 or less 
lessons) 

80% 

21% 

No 
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(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 
(0-7 across all 
lessons)) 







C.A.M.P. 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 3 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-5 across all 
lessons)) 

14 

treatment 

teachers 

S 

Adequate: >= 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 
teachers cover 3 or less 
lessons) 

80% 

21% 

No 

C.A.M.P. 

College 

Awareness 

Curriculum 

Teacher implements 
RTF college 
awareness 

curriculum as trained 

in PD: Units 1-3 

14 

treatment 

teachers 

S 

All three CAC units are 
presented in treatment 
classrooms 

80% 

79% 

No 

Algebra 1 

Curricular 

Materials 

Teacher receives all 
curriculum materials 

16 

treatment 

teachers 

S 

Received materials / No 
or Yes: 0-1 

100% 

100% 

Yes 
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for intervention 







Algebra 1 

Professional 

Development 

Teacher attends 
training for 
implementing 
ConnectEd Algebra 1 
curriculum 

Teacher attends 
training for 
implementing RTF 
college awareness 
curriculum 

16 

treatment 

teachers 

S 

Teachers participated in 
all 5 days of PD for 
Summer C.A.M.P. 

AND 

participated in all four 
hours of CAC training. 

80% 

88% 

Yes 

Algebra 1 
Coaching 

Y1 Algebra 1 

Coaching and 
Collaboration for all 

treatment teachers 
(online sessions 
scheduled for 1-3 
hours/week): 

Units 1-3 

16 

treatment 

teachers 

S 

Teachers participated in 
a minimum of one hour 
of Coaching/ 
Collaboration time per 
week for EACH unit 
(units 1-3) 

80% 

81% 

Yes 

Algebra 1 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 1 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 

16 

treatment 

teachers 

S 

Adequate: > 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 
teachers cover 3 or less 
lessons) 

80% 

94% 

Yes 
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attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 
(0-5 across all 
lessons)) 







Algebra 1 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 2 

a. Number of lessons 
used (> 72% of 
implementation range 
i.e., 5 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 
lesson / No or Yes: 0- 
1 for each taught 
lesson (0-7 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-7 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-7 across all 
lessons)) 

16 

treatment 

teachers 

S 

Adequate: > 72% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 16 
(this indicates that the 
minimal number of 
lessons to be taught is 

5; no way to reach a 
score of 16 or above if 
teachers cover 4 or less 
lessons) 

80% 

81% 

Yes 

Algebra 1 

Teacher 
Implementation 
of Classroom 
Intervention: 

Unit 3 

a. Number of lessons 
used (> 80% of 
implementation range 
i.e., 4 lessons 

/ No or Yes: 0-1) 

b. Time spent on 
each lesson (> 70% 
of implementation 
range for each taught 

16 

treatment 

teachers 

S 

Adequate: > 75% of 
possible highest 
composite score; i.e., 
the score for 
implementation with 
fidelity is at least 12 
(this indicates that the 
minimal number of 
lessons to be taught is 

4; no way to reach a 
score of 12 or above if 

80% 

50% 

No 
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lesson / No or Yes: 0- 
1 for each taught 
lesson (0-5 across all 
lessons)) 

c. Number of steps 
used within each 
lesson ( > 70% of 
implementation range 
for each taught 
lesson 

/ No or Yes: 0-1 for 
each taught lesson 
(0-5 across all 
lessons)) 

d. Average level of 
attention given to 
each lesson 
(moderate level of 
attention for each 
taught lesson / No or 
Yes: 0-1 for each 
taught lesson 

(0-5 across all 
lessons)) 



teachers cover 3 or less 
lessons) 




Algebra 1 

College 

Awareness 

Curriculum 

Teacher implements 
RTF college 
awareness 

curriculum (2 
activities) as trained 
in PD: Units 1-3 

16 

treatment 

teachers 

S 

Two activities (fora 
maximum of 30 
minutes) are 
implemented in the 
treatment classrooms. 

80% 

13% 

No 

a Mediators refer to measurement of short-term outcomes, e.g., changes in student/teacher/parent behaviors or attitudes that are 
assumed to have a direct connection to long-term outcomes. If these mediators have been measured in both the treatment and 
comparison groups and are being reported as part of an exploratory contrast, they should not be entered here. If these mediators 
have been measured in the treatment group only or in both the treatment and the comparison group and are not being reported as 
part of an exploratory contrast, the findings for just the treatment group would be reported here. If evaluators believe they have 
findings on mediators to report, they should consult with their TA Liaison or the AR helpdesk to confirm that the findings are 
appropriate for this section of the Survey. 

b All: If the intervention group in the impact analysis includes 10 schools and fidelity measurement includes these 10 schools, the 
evaluator would enter “A” indicating that All of the schools in the impact analysis are represented in the fidelity findings. Some: If the 
intervention group in the impact analysis includes teachers in grades K to 3 but fidelity is measured only for teachers in Kindergarten, 
the evaluator would enter “S” indicating that Some of the teachers in the impact analysis are represented in the fidelity findings. None: 

If the intervention group in the impact analysis includes grades 7 - 9 but fidelity is measured only for grades 5-6, the evaluator would 
enter “N” indicating that None of the grades in the impact analysis are represented in the fidelity findings. 
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